By type
+ CONSUL_BIND=
+ [ -n ]
+ CONSUL_CLIENT=
+ [ -n ]
+ CONSUL_DATA_DIR=/consul/data
+ CONSUL_CONFIG_DIR=/consul/config
+ [ -n ]
+ [ a = - ]
+ [ agent = agent ]
+ shift
+ set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
+ [ consul = consul ]
+ stat -c %u /consul/data
+ id -u consul
+ [ 100 != 100 ]
+ stat -c %u /consul/config
+ id -u consul
+ [ 100 != 100 ]
+ [ ! -z ]
+ exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
bootstrap_expect > 0: expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
Version: 'v1.0.6'
Node ID: 'b8761040-27af-3006-8f09-909b0e3596a3'
Node name: 'dev-consul-server-0'
Datacenter: 'dc1' (Segment: '')
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.242.228.137 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2021/04/14 10:40:13 [INFO] raft: Initial configuration (index=0): []
2021/04/14 10:40:13 [INFO] raft: Node at 10.242.228.137:8300 [Follower] entering Follower state (Leader: "")
2021/04/14 10:40:13 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.228.137
2021/04/14 10:40:14 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.228.137
2021/04/14 10:40:14 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.228.137:8300) (DC: dc1)
2021/04/14 10:40:14 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/04/14 10:40:14 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2021/04/14 10:40:14 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2021/04/14 10:40:14 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
2021/04/14 10:40:14 [INFO] agent: started state syncer
2021/04/14 10:40:14 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer
2021/04/14 10:40:14 [INFO] agent: Joining LAN cluster...
2021/04/14 10:40:14 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/04/14 10:40:14 [WARN] memberlist: Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
2021/04/14 10:40:14 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
2021/04/14 10:40:14 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/04/14 10:40:14 [INFO] agent: (LAN) joined: 0 Err: 3 error(s) occurred:
* Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/04/14 10:40:14 [WARN] agent: Join LAN failed: , retrying in 30s
2021/04/14 10:40:21 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 10:40:21 [WARN] raft: no known peers, aborting election
==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=783ed15a-60dd-b03d-8b45-e381887955ce&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2021/04/14 10:40:44 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-rfzpr 10.242.228.136
2021/04/14 10:40:44 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/04/14 10:40:44 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
2021/04/14 10:40:44 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/04/14 10:40:44 [INFO] agent: (LAN) joined: 1 Err:
2021/04/14 10:40:44 [INFO] agent: Join LAN completed. Synced with 1 initial agents
2021/04/14 10:40:47 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 10:40:48 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 10:41:16 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 10:41:16 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 10:41:39 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 10:41:42 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 10:41:43 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 10:41:43 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 10:41:43 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 10:41:43 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 10:42:07 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 10:42:12 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 10:42:39 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 10:42:42 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 10:43:04 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 10:43:14 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 10:43:22 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.70.73
2021/04/14 10:43:22 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 10:43:22 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.242.228.137:8300,10.242.5.10:8300,10.242.70.73:8300
2021/04/14 10:43:22 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.70.73
2021/04/14 10:43:22 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/14 10:43:24 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/14 10:43:24 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 2
2021/04/14 10:43:24 [INFO] raft: Election won. Tally: 2
2021/04/14 10:43:24 [INFO] raft: Node at 10.242.228.137:8300 [Leader] entering Leader state
2021/04/14 10:43:24 [INFO] raft: Added peer 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, starting replication
2021/04/14 10:43:24 [INFO] raft: Added peer 590cff07-d5aa-e855-db2c-f428e810df3b, starting replication
2021/04/14 10:43:24 [INFO] consul: cluster leadership acquired
2021/04/14 10:43:24 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/14 10:43:24 [WARN] raft: AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300} rejected, sending older logs (next: 1)
2021/04/14 10:43:24 [WARN] raft: AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300} rejected, sending older logs (next: 1)
2021/04/14 10:43:24 [INFO] raft: pipelining replication to peer {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}
2021/04/14 10:43:24 [INFO] raft: pipelining replication to peer {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}
2021/04/14 10:43:24 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/04/14 10:43:24 [INFO] consul: member 'dev-consul-7467bf7b46-rfzpr' joined, marking health alive
2021/04/14 10:43:24 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/14 10:43:24 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/14 10:43:25 [INFO] agent: Synced node info
2021/04/14 11:00:59 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:00:59 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:00:59 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/14 11:01:03 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:01:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:45462->10.242.5.10:8301: i/o timeout
2021/04/14 11:01:05 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:01:05 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 2.500277463s
2021/04/14 11:01:05 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/14 11:01:09 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:01:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:45578->10.242.5.10:8301: i/o timeout
2021/04/14 11:01:11 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:01:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:13 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 2.500313415s
2021/04/14 11:01:13 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:01:13 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:01:13 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/14 11:01:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:45612->10.242.5.10:8301: i/o timeout
2021/04/14 11:01:14 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:01:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:15 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 4.993283812s
2021/04/14 11:01:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:18 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 7.452498395s
2021/04/14 11:01:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:47342->10.242.5.10:8302: i/o timeout
2021/04/14 11:01:18 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:01:19 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:46180->10.242.5.10:8300: i/o timeout
2021/04/14 11:01:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:20 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:01:21 [INFO] raft: aborting pipeline replication to peer {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}
2021/04/14 11:01:21 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:01:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:23 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 2.50938225s
2021/04/14 11:01:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:24 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:01:24 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/14 11:01:26 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 4.966959003s
2021/04/14 11:01:27 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:01:27 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:01:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:28 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 7.454233014s
2021/04/14 11:01:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:47542->10.242.5.10:8302: i/o timeout
2021/04/14 11:01:28 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:01:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:30 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:35410->10.242.5.10:8300: i/o timeout
2021/04/14 11:01:31 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:35418->10.242.5.10:8300: i/o timeout
2021/04/14 11:01:31 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:01:31 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:01:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:47742->10.242.5.10:8302: i/o timeout
2021/04/14 11:01:38 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:01:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:41 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:35602->10.242.5.10:8300: i/o timeout
2021/04/14 11:01:41 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:01:41 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:35612->10.242.5.10:8300: i/o timeout
2021/04/14 11:01:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:42 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:01:44 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:01:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:48 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:01:48 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 11:01:48 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 11:01:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:51 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:35804->10.242.5.10:8300: i/o timeout
2021/04/14 11:01:51 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:01:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:52 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:35832->10.242.5.10:8300: i/o timeout
2021/04/14 11:01:52 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:01:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:48052->10.242.5.10:8302: i/o timeout
2021/04/14 11:01:53 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:01:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:01:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:01 [ERR] memberlist: failed to receive: read tcp 10.242.228.137:8301->10.242.5.10:39474: i/o timeout from=10.242.5.10:39474
2021/04/14 11:02:01 [ERR] memberlist: Failed to send error: write tcp 10.242.228.137:8301->10.242.5.10:39474: i/o timeout from=10.242.5.10:39474
2021/04/14 11:02:01 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:36010->10.242.5.10:8300: i/o timeout
2021/04/14 11:02:01 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:02:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:02 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:36056->10.242.5.10:8300: i/o timeout
2021/04/14 11:02:03 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:02:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:06 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:02:06 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/14 11:02:09 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:02:09 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.5.10:53592
2021/04/14 11:02:09 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:02:09 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:02:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:11 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:36210->10.242.5.10:8300: i/o timeout
2021/04/14 11:02:11 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:02:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:13 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:36260->10.242.5.10:8300: i/o timeout
2021/04/14 11:02:14 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.5.10:8302
2021/04/14 11:02:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:14 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:02:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:18 [ERR] memberlist: failed to receive: read tcp 10.242.228.137:8302->10.242.5.10:55986: i/o timeout from=10.242.5.10:55986
2021/04/14 11:02:18 [ERR] memberlist: Failed to send error: write tcp 10.242.228.137:8302->10.242.5.10:55986: i/o timeout from=10.242.5.10:55986
2021/04/14 11:02:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:21 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:36416->10.242.5.10:8300: i/o timeout
2021/04/14 11:02:21 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:02:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:24 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:02:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:24 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:36484->10.242.5.10:8300: i/o timeout
2021/04/14 11:02:25 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:02:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:31 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:36616->10.242.5.10:8300: i/o timeout
2021/04/14 11:02:31 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:02:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:35 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:36702->10.242.5.10:8300: i/o timeout
2021/04/14 11:02:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:36 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:02:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:41 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:36822->10.242.5.10:8300: i/o timeout
2021/04/14 11:02:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:42 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:02:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:46 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:36918->10.242.5.10:8300: i/o timeout
2021/04/14 11:02:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:48 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:02:48 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/14 11:02:48 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:02:51 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:02:51 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:02:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:52 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:37054->10.242.5.10:8300: i/o timeout
2021/04/14 11:02:53 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:02:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:02:58 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:37168->10.242.5.10:8300: i/o timeout
2021/04/14 11:03:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:02 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:03:03 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:37268->10.242.5.10:8300: i/o timeout
2021/04/14 11:03:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:06 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:03:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:10 [DEBUG] raft-net: 10.242.228.137:8300 accepted connection from: 10.242.5.10:40348
2021/04/14 11:03:10 [WARN] raft: Rejecting vote request from 10.242.5.10:8300 since we have a leader: 10.242.228.137:8300
2021/04/14 11:03:10 [ERR] raft: peer {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300} has newer term, stopping replication
2021/04/14 11:03:10 [INFO] raft: Node at 10.242.228.137:8300 [Follower] entering Follower state (Leader: "")
2021/04/14 11:03:10 [INFO] raft: aborting pipeline replication to peer {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}
2021/04/14 11:03:10 [INFO] consul: cluster leadership lost
2021/04/14 11:03:10 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 11:03:10 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 11:03:10 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:03:10 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:03:11 [WARN] consul.coordinate: Batch update failed: node is not the leader
2021/04/14 11:03:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:48130->10.242.5.10:8301: i/o timeout
2021/04/14 11:03:18 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:03:19 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/14 11:03:19 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 3
2021/04/14 11:03:19 [INFO] raft: Election won. Tally: 2
2021/04/14 11:03:19 [INFO] raft: Node at 10.242.228.137:8300 [Leader] entering Leader state
2021/04/14 11:03:19 [INFO] raft: Added peer 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, starting replication
2021/04/14 11:03:19 [INFO] raft: Added peer 590cff07-d5aa-e855-db2c-f428e810df3b, starting replication
2021/04/14 11:03:19 [INFO] consul: cluster leadership acquired
2021/04/14 11:03:19 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/14 11:03:19 [INFO] raft: pipelining replication to peer {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}
2021/04/14 11:03:19 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/14 11:03:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:48194->10.242.5.10:8301: i/o timeout
2021/04/14 11:03:21 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:03:21 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/14 11:03:21 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:03:21 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:03:21 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/14 11:03:21 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 2.502319073s
2021/04/14 11:03:22 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:03:22 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:03:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:49862->10.242.5.10:8302: i/o timeout
2021/04/14 11:03:23 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:03:24 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 5.003046445s
2021/04/14 11:03:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:26 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 7.488033818s
2021/04/14 11:03:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:29 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:37520->10.242.5.10:8300: i/o timeout
2021/04/14 11:03:29 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:37444->10.242.5.10:8300: i/o timeout
2021/04/14 11:03:29 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:03:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:30 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:03:30 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/14 11:03:30 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:37810->10.242.5.10:8300: i/o timeout
2021/04/14 11:03:30 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:03:32 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:03:32 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:03:32 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/14 11:03:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:33 [WARN] raft: Rejecting vote request from 10.242.5.10:8300 since we have a leader: 10.242.228.137:8300
2021/04/14 11:03:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:50150->10.242.5.10:8302: i/o timeout
2021/04/14 11:03:38 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:03:39 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:37978->10.242.5.10:8300: i/o timeout
2021/04/14 11:03:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:39 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:03:40 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:38018->10.242.5.10:8300: i/o timeout
2021/04/14 11:03:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:41 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:03:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:50256->10.242.5.10:8302: i/o timeout
2021/04/14 11:03:48 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:03:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:49 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:38174->10.242.5.10:8300: i/o timeout
2021/04/14 11:03:49 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:03:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:51 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:38224->10.242.5.10:8300: i/o timeout
2021/04/14 11:03:52 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:03:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:53 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:03:53 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 11:03:53 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 11:03:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:03:59 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:38392->10.242.5.10:8300: i/o timeout
2021/04/14 11:03:59 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:04:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:04:02 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:38462->10.242.5.10:8300: i/o timeout
2021/04/14 11:04:03 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:04:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:04:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:04:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:04:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:04:09 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:38592->10.242.5.10:8300: i/o timeout
2021/04/14 11:04:09 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:04:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:04:13 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:38690->10.242.5.10:8300: i/o timeout
2021/04/14 11:04:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:04:13 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:04:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:04:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:04:18 [ERR] memberlist: failed to receive: read tcp 10.242.228.137:8301->10.242.5.10:40588: i/o timeout from=10.242.5.10:40588
2021/04/14 11:04:18 [ERR] memberlist: Failed to send error: write tcp 10.242.228.137:8301->10.242.5.10:40588: i/o timeout from=10.242.5.10:40588
2021/04/14 11:04:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:04:19 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:38800->10.242.5.10:8300: i/o timeout
2021/04/14 11:04:19 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:04:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:04:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:04:23 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:38900->10.242.5.10:8300: i/o timeout
2021/04/14 11:04:24 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:04:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:04:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:04:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:04:29 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:39010->10.242.5.10:8300: i/o timeout
2021/04/14 11:04:30 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:04:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:04:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:04:34 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:04:34 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:39122->10.242.5.10:8300: i/o timeout
2021/04/14 11:04:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:04:35 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:04:37 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:04:37 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:04:37 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/14 11:04:37 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 11:04:37 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 11:04:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:04:38 [WARN] raft: Rejecting vote request from 10.242.5.10:8300 since we have a leader: 10.242.228.137:8300
2021/04/14 11:04:38 [ERR] raft: peer {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300} has newer term, stopping replication
2021/04/14 11:04:38 [INFO] raft: Node at 10.242.228.137:8300 [Follower] entering Follower state (Leader: "")
2021/04/14 11:04:38 [INFO] raft: aborting pipeline replication to peer {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}
2021/04/14 11:04:38 [INFO] consul: cluster leadership lost
2021/04/14 11:04:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:49844->10.242.5.10:8301: i/o timeout
2021/04/14 11:04:41 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:04:43 [DEBUG] raft-net: 10.242.228.137:8300 accepted connection from: 10.242.70.73:41058
2021/04/14 11:04:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:49916->10.242.5.10:8301: i/o timeout
2021/04/14 11:04:44 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:04:44 [INFO] consul: New leader elected: dev-consul-server-2
2021/04/14 11:04:44 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:04:44 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:04:44 [DEBUG] raft-net: 10.242.228.137:8300 accepted connection from: 10.242.70.73:41070
2021/04/14 11:04:51 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:04:51 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:04:53 [WARN] raft: Rejecting vote request from 10.242.5.10:8300 since we have a leader: 10.242.70.73:8300
2021/04/14 11:04:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:50122->10.242.5.10:8301: i/o timeout
2021/04/14 11:04:54 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:04:54 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/04/14 11:04:56 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/14 11:04:56 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:04:56 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:04:56 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:04:56 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:05:02 [WARN] raft: Heartbeat timeout from "10.242.70.73:8300" reached, starting election
2021/04/14 11:05:02 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 5
2021/04/14 11:05:02 [INFO] raft: Node at 10.242.228.137:8300 [Follower] entering Follower state (Leader: "")
2021/04/14 11:05:03 [WARN] raft: Rejecting vote request from 10.242.5.10:8300 since our last term is greater (4, 2)
2021/04/14 11:05:03 [INFO] consul: New leader elected: dev-consul-server-2
2021/04/14 11:05:04 [DEBUG] raft-net: 10.242.228.137:8300 accepted connection from: 10.242.70.73:41284
2021/04/14 11:06:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:52148->10.242.5.10:8301: i/o timeout
2021/04/14 11:06:30 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:06:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:52196->10.242.5.10:8301: i/o timeout
2021/04/14 11:06:33 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:06:34 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/14 11:06:34 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:06:34 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:06:38 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:06:38 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:06:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:52352->10.242.5.10:8301: i/o timeout
2021/04/14 11:06:40 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:06:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:52494->10.242.5.10:8301: i/o timeout
2021/04/14 11:06:46 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:06:46 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/14 11:07:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:52918->10.242.5.10:8301: i/o timeout
2021/04/14 11:07:07 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:07:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:52972->10.242.5.10:8301: i/o timeout
2021/04/14 11:07:10 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:07:11 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:07:11 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:07:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:54786->10.242.5.10:8302: i/o timeout
2021/04/14 11:07:18 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:07:27 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:07:27 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.5.10:42498
2021/04/14 11:07:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:55100->10.242.5.10:8302: i/o timeout
2021/04/14 11:07:33 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:07:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:55290->10.242.5.10:8302: i/o timeout
2021/04/14 11:07:43 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:07:47 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:07:47 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 11:07:47 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 11:08:14 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.5.10:8302
2021/04/14 11:09:07 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:09:44 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.5.10:8302
2021/04/14 11:10:17 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:10:17 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:11:14 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.5.10:8302
2021/04/14 11:11:57 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:11:58 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/04/14 11:12:14 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.5.10:8302
2021/04/14 11:12:35 [WARN] raft: Rejecting vote request from 10.242.5.10:8300 since we have a leader: 10.242.70.73:8300
2021/04/14 11:12:39 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 11:12:39 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 11:12:39 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:12:39 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:12:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:60090->10.242.5.10:8301: i/o timeout
2021/04/14 11:12:42 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:12:45 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/04/14 11:12:45 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:12:45 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:12:45 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/14 11:12:48 [INFO] consul: New leader elected: dev-consul-server-2
2021/04/14 11:12:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:33640->10.242.5.10:8302: i/o timeout
2021/04/14 11:12:48 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:12:49 [DEBUG] raft-net: 10.242.228.137:8300 accepted connection from: 10.242.70.73:46134
2021/04/14 11:13:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:33964->10.242.5.10:8302: i/o timeout
2021/04/14 11:13:03 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:13:12 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:13:12 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:13:12 [WARN] raft: Rejecting vote request from 10.242.5.10:8300 since we have a leader: 10.242.70.73:8300
2021/04/14 11:13:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:60844->10.242.5.10:8301: i/o timeout
2021/04/14 11:13:15 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:13:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:60880->10.242.5.10:8301: i/o timeout
2021/04/14 11:13:17 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:13:17 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/14 11:13:17 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:13:17 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:13:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:34294->10.242.5.10:8302: i/o timeout
2021/04/14 11:13:18 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:13:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:34630->10.242.5.10:8302: i/o timeout
2021/04/14 11:13:33 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:13:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:34828->10.242.5.10:8302: i/o timeout
2021/04/14 11:13:43 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:13:44 [ERR] memberlist: failed to receive: read tcp 10.242.228.137:8301->10.242.5.10:45524: i/o timeout from=10.242.5.10:45524
2021/04/14 11:13:44 [ERR] memberlist: Failed to send error: write tcp 10.242.228.137:8301->10.242.5.10:45524: i/o timeout from=10.242.5.10:45524
2021/04/14 11:13:48 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:13:48 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 11:13:48 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 11:13:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:34934->10.242.5.10:8302: i/o timeout
2021/04/14 11:13:53 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:14:07 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:14:14 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:14:39 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.5.10:8302
2021/04/14 11:15:03 [ERR] memberlist: failed to receive: read tcp 10.242.228.137:8301->10.242.5.10:46214: i/o timeout from=10.242.5.10:46214
2021/04/14 11:15:03 [ERR] memberlist: Failed to send error: write tcp 10.242.228.137:8301->10.242.5.10:46214: i/o timeout from=10.242.5.10:46214
2021/04/14 11:15:15 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 11:15:15 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 11:15:15 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:15:15 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:15:16 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/14 11:15:16 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-rfzpr 10.242.228.136
2021/04/14 11:15:17 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/04/14 11:15:17 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-rfzpr 10.242.228.136
2021/04/14 11:15:22 [WARN] raft: Rejecting vote request from 10.242.5.10:8300 since we have a leader: 10.242.70.73:8300
2021/04/14 11:15:23 [INFO] consul: New leader elected: dev-consul-server-2
2021/04/14 11:15:24 [DEBUG] raft-net: 10.242.228.137:8300 accepted connection from: 10.242.70.73:48348
2021/04/14 11:15:49 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/14 11:16:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:36008->10.242.5.10:8301: i/o timeout
2021/04/14 11:16:00 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:16:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:36524->10.242.5.10:8301: i/o timeout
2021/04/14 11:16:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:36630->10.242.5.10:8301: i/o timeout
2021/04/14 11:16:30 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:16:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:36714->10.242.5.10:8301: i/o timeout
2021/04/14 11:16:33 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:16:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:36884->10.242.5.10:8301: i/o timeout
2021/04/14 11:16:42 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:16:44 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/14 11:16:44 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:16:44 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:16:44 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/04/14 11:16:44 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:16:44 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:16:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:36998->10.242.5.10:8301: i/o timeout
2021/04/14 11:16:47 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:16:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:37196->10.242.5.10:8301: i/o timeout
2021/04/14 11:16:56 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:17:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:37268->10.242.5.10:8301: i/o timeout
2021/04/14 11:17:00 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:17:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:37552->10.242.5.10:8301: i/o timeout
2021/04/14 11:17:13 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:17:13 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/14 11:17:13 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:17:13 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:17:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:37580->10.242.5.10:8301: i/o timeout
2021/04/14 11:17:15 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:17:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:39268->10.242.5.10:8302: i/o timeout
2021/04/14 11:17:20 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:17:20 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:17:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:37822->10.242.5.10:8301: i/o timeout
2021/04/14 11:17:26 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:17:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:37884->10.242.5.10:8301: i/o timeout
2021/04/14 11:17:29 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:17:30 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/14 11:17:30 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:17:30 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:17:44 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:17:44 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:17:44 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/14 11:18:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:38684->10.242.5.10:8301: i/o timeout
2021/04/14 11:18:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:54050->10.242.70.73:8301: i/o timeout
2021/04/14 11:18:19 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:18:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:54112->10.242.70.73:8301: i/o timeout
2021/04/14 11:18:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:18:23 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.70.73
2021/04/14 11:18:23 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:18:25 [WARN] raft: Heartbeat timeout from "10.242.70.73:8300" reached, starting election
2021/04/14 11:18:25 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 13
2021/04/14 11:18:25 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:18:25 [INFO] raft: Duplicate RequestVote for same term: 13
2021/04/14 11:18:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:57436->10.242.70.73:8302: i/o timeout
2021/04/14 11:18:28 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/14 11:18:31 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:18:31 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 14
2021/04/14 11:18:31 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:18:31 [INFO] raft: Election won. Tally: 2
2021/04/14 11:18:31 [INFO] raft: Node at 10.242.228.137:8300 [Leader] entering Leader state
2021/04/14 11:18:31 [INFO] raft: Added peer 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, starting replication
2021/04/14 11:18:31 [INFO] raft: Added peer 590cff07-d5aa-e855-db2c-f428e810df3b, starting replication
2021/04/14 11:18:31 [INFO] consul: cluster leadership acquired
2021/04/14 11:18:31 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/14 11:18:31 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:18:31 [INFO] raft: pipelining replication to peer {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}
2021/04/14 11:18:31 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/14 11:18:32 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:18:33 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 2.502149118s
2021/04/14 11:18:34 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:18:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:18:35 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:57840->10.242.70.73:8300: i/o timeout
2021/04/14 11:18:36 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 4.992792299s
2021/04/14 11:18:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:18:38 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 7.445466951s
2021/04/14 11:18:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:57648->10.242.70.73:8302: i/o timeout
2021/04/14 11:18:38 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/14 11:18:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:18:41 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:48842->10.242.70.73:8300: i/o timeout
2021/04/14 11:18:41 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:48844->10.242.70.73:8300: i/o timeout
2021/04/14 11:18:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:18:41 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:18:42 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: read tcp 10.242.228.137:48866->10.242.70.73:8300: i/o timeout
2021/04/14 11:18:42 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:18:44 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:18:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:18:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:18:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:57878->10.242.70.73:8302: i/o timeout
2021/04/14 11:18:48 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/14 11:18:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:18:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:18:51 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:49060->10.242.70.73:8300: i/o timeout
2021/04/14 11:18:51 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:18:52 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: read tcp 10.242.228.137:49100->10.242.70.73:8300: i/o timeout
2021/04/14 11:18:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:18:53 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:18:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:18:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:18:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:57994->10.242.70.73:8302: i/o timeout
2021/04/14 11:18:58 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/14 11:18:58 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:18:58 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.70.73
2021/04/14 11:18:58 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/14 11:18:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:19:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:19:01 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:49296->10.242.70.73:8300: i/o timeout
2021/04/14 11:19:01 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:19:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:19:03 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: read tcp 10.242.228.137:49362->10.242.70.73:8300: i/o timeout
2021/04/14 11:19:04 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:19:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:19:06 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:19:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:19:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:19:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:40010->10.242.5.10:8301: i/o timeout
2021/04/14 11:19:09 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:19:09 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 2.500289775s
2021/04/14 11:19:09 [WARN] raft: Failed to contact quorum of nodes, stepping down
2021/04/14 11:19:09 [INFO] raft: Node at 10.242.228.137:8300 [Follower] entering Follower state (Leader: "")
2021/04/14 11:19:09 [INFO] raft: aborting pipeline replication to peer {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}
2021/04/14 11:19:09 [WARN] consul.coordinate: Batch update failed: leadership lost while committing log
2021/04/14 11:19:09 [INFO] consul: cluster leadership lost
2021/04/14 11:19:09 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.70.73:8302
2021/04/14 11:19:09 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:19:09 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.70.73:56302
2021/04/14 11:19:09 [ERR] memberlist: failed to receive: read tcp 10.242.228.137:8301->10.242.70.73:36170: i/o timeout from=10.242.70.73:36170
2021/04/14 11:19:09 [ERR] memberlist: Failed to send error: write tcp 10.242.228.137:8301->10.242.70.73:36170: i/o timeout from=10.242.70.73:36170
2021/04/14 11:19:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:40062->10.242.5.10:8301: i/o timeout
2021/04/14 11:19:11 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:19:11 [DEBUG] raft-net: 10.242.228.137:8300 accepted connection from: 10.242.70.73:51128
2021/04/14 11:19:11 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:49502->10.242.70.73:8300: i/o timeout
2021/04/14 11:19:11 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.70.73
2021/04/14 11:19:11 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:19:12 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.70.73
2021/04/14 11:19:12 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/14 11:19:13 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:19:13 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:19:13 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:19:14 [WARN] consul.coordinate: Batch update failed: node is not the leader
2021/04/14 11:19:14 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:19:14 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:19:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:55312->10.242.70.73:8301: i/o timeout
2021/04/14 11:19:16 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/14 11:19:16 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 15
2021/04/14 11:19:16 [INFO] raft: Election won. Tally: 2
2021/04/14 11:19:16 [INFO] raft: Node at 10.242.228.137:8300 [Leader] entering Leader state
2021/04/14 11:19:16 [INFO] raft: Added peer 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, starting replication
2021/04/14 11:19:16 [INFO] raft: Added peer 590cff07-d5aa-e855-db2c-f428e810df3b, starting replication
2021/04/14 11:19:16 [INFO] consul: cluster leadership acquired
2021/04/14 11:19:16 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/14 11:19:16 [WARN] raft: AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300} rejected, sending older logs (next: 507)
2021/04/14 11:19:16 [INFO] raft: pipelining replication to peer {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}
2021/04/14 11:19:16 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/14 11:19:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:40228->10.242.5.10:8301: i/o timeout
2021/04/14 11:19:18 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:19:18 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 2.502852679s
2021/04/14 11:19:19 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:19:19 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:19:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:19:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:40300->10.242.5.10:8301: i/o timeout
2021/04/14 11:19:21 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:19:21 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 4.9581556s
2021/04/14 11:19:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:19:22 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 6.627490264s
2021/04/14 11:19:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:40346->10.242.5.10:8301: i/o timeout
2021/04/14 11:19:23 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:19:23 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:19:23 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:19:23 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/14 11:19:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:19:24 [INFO] raft: pipelining replication to peer {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}
2021/04/14 11:19:24 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:19:24 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:19:24 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:19:24 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/14 11:19:29 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:19:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:55624->10.242.70.73:8301: i/o timeout
2021/04/14 11:19:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:19:35 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:19:36 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/14 11:19:38 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/14 11:19:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:55846->10.242.70.73:8301: i/o timeout
2021/04/14 11:19:40 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:19:41 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:19:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:19:44 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/14 11:19:44 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.70.73
2021/04/14 11:19:44 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:19:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:55952->10.242.70.73:8301: i/o timeout
2021/04/14 11:19:44 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/14 11:19:44 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:19:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:19:45 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 2.50024613s
2021/04/14 11:19:46 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.70.73
2021/04/14 11:19:46 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:19:46 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/14 11:19:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:19:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:56028->10.242.70.73:8301: i/o timeout
2021/04/14 11:19:48 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:19:48 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 2.500254609s
2021/04/14 11:19:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:19:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:19:50 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 4.940504338s
2021/04/14 11:19:50 [INFO] raft: aborting pipeline replication to peer {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}
2021/04/14 11:19:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:56100->10.242.70.73:8301: i/o timeout
2021/04/14 11:19:51 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:19:52 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/14 11:19:52 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.70.73
2021/04/14 11:19:52 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:19:52 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/14 11:19:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:19:53 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 2.500279595s
2021/04/14 11:19:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:19:55 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:19:55 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 4.960424184s
2021/04/14 11:19:56 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: read tcp 10.242.228.137:49864->10.242.70.73:8300: i/o timeout
2021/04/14 11:19:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:19:56 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:19:56 [INFO] raft: pipelining replication to peer {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}
2021/04/14 11:19:56 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.70.73
2021/04/14 11:19:56 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:19:56 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/14 11:20:07 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:20:07 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:20:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:41406->10.242.5.10:8301: i/o timeout
2021/04/14 11:20:11 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:20:11 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:20:11 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/14 11:20:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:20:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:20:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:41494->10.242.5.10:8301: i/o timeout
2021/04/14 11:20:15 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:20:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:20:16 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 2.50135268s
2021/04/14 11:20:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:20:18 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 4.939379031s
2021/04/14 11:20:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:41578->10.242.5.10:8301: i/o timeout
2021/04/14 11:20:19 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:20:19 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/14 11:20:19 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:20:19 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:20:19 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/14 11:20:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:20:21 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 7.431745646s
2021/04/14 11:20:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:20:24 [INFO] raft: aborting pipeline replication to peer {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}
2021/04/14 11:20:24 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:20:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:20:24 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:39340->10.242.5.10:8300: i/o timeout
2021/04/14 11:20:25 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:20:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:20:26 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 2.5002616s
2021/04/14 11:20:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:20:29 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 4.957193552s
2021/04/14 11:20:30 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/14 11:20:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:20:30 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:20:30 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.5.10:48866
2021/04/14 11:20:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:56970->10.242.70.73:8301: i/o timeout
2021/04/14 11:20:31 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:20:31 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:20:31 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/14 11:20:31 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 7.453274685s
2021/04/14 11:20:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:20:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:20:33 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 2.500340456s
2021/04/14 11:20:33 [WARN] raft: Failed to contact quorum of nodes, stepping down
2021/04/14 11:20:33 [INFO] raft: Node at 10.242.228.137:8300 [Follower] entering Follower state (Leader: "")
2021/04/14 11:20:33 [INFO] raft: aborting pipeline replication to peer {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}
2021/04/14 11:20:33 [INFO] consul: cluster leadership lost
2021/04/14 11:20:33 [WARN] consul.coordinate: Batch update failed: leadership lost while committing log
2021/04/14 11:20:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:43520->10.242.5.10:8302: i/o timeout
2021/04/14 11:20:33 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:20:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:57062->10.242.70.73:8301: i/o timeout
2021/04/14 11:20:34 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:20:34 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:58072->10.242.5.10:8300: i/o timeout
2021/04/14 11:20:35 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:59588->10.242.5.10:8300: i/o timeout
2021/04/14 11:20:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:57082->10.242.70.73:8301: i/o timeout
2021/04/14 11:20:36 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:20:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:57158->10.242.70.73:8301: i/o timeout
2021/04/14 11:20:39 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:20:41 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/14 11:20:41 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 16
2021/04/14 11:20:41 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:20:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:57206->10.242.70.73:8301: i/o timeout
2021/04/14 11:20:42 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:20:43 [INFO] raft: Duplicate RequestVote for same term: 16
2021/04/14 11:20:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:43628->10.242.5.10:8302: i/o timeout
2021/04/14 11:20:43 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:20:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:57310->10.242.70.73:8301: i/o timeout
2021/04/14 11:20:45 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:20:47 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:20:47 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:20:47 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/14 11:20:47 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/14 11:20:47 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.70.73
2021/04/14 11:20:47 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:20:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:57338->10.242.70.73:8301: i/o timeout
2021/04/14 11:20:48 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:20:48 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/04/14 11:20:48 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:53982
2021/04/14 11:20:49 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:20:49 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 17
2021/04/14 11:20:49 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:20:49 [INFO] raft: Duplicate RequestVote for same term: 17
2021/04/14 11:20:49 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:51146
2021/04/14 11:20:51 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:50772->10.242.70.73:8300: i/o timeout
2021/04/14 11:20:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:42304->10.242.5.10:8301: i/o timeout
2021/04/14 11:20:52 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:20:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:42354->10.242.5.10:8301: i/o timeout
2021/04/14 11:20:54 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:20:55 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:20:55 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.70.73:51148
2021/04/14 11:20:55 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:20:55 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 18
2021/04/14 11:20:55 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:20:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:42438->10.242.5.10:8301: i/o timeout
2021/04/14 11:20:58 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:20:58 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:54134
2021/04/14 11:20:58 [INFO] raft: Election won. Tally: 2
2021/04/14 11:20:58 [INFO] raft: Node at 10.242.228.137:8300 [Leader] entering Leader state
2021/04/14 11:20:58 [INFO] raft: Added peer 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, starting replication
2021/04/14 11:20:58 [INFO] raft: Added peer 590cff07-d5aa-e855-db2c-f428e810df3b, starting replication
2021/04/14 11:20:58 [INFO] consul: cluster leadership acquired
2021/04/14 11:20:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:60560->10.242.70.73:8302: i/o timeout
2021/04/14 11:20:58 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/14 11:20:58 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/14 11:20:58 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:20:59 [WARN] raft: AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300} rejected, sending older logs (next: 525)
2021/04/14 11:20:59 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:51918->10.242.70.73:8300: i/o timeout
2021/04/14 11:20:59 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:21:00 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/04/14 11:21:01 [INFO] raft: pipelining replication to peer {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}
2021/04/14 11:21:01 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/14 11:21:01 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/14 11:21:01 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 2.50691107s
2021/04/14 11:21:03 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 4.76331498s
2021/04/14 11:21:04 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:21:04 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:21:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:42564->10.242.5.10:8301: i/o timeout
2021/04/14 11:21:05 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:21:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:21:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:21:05 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:52044->10.242.70.73:8300: i/o timeout
2021/04/14 11:21:05 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 6.793460349s
2021/04/14 11:21:05 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 2.500411525s
2021/04/14 11:21:05 [WARN] raft: Failed to contact quorum of nodes, stepping down
2021/04/14 11:21:05 [INFO] raft: Node at 10.242.228.137:8300 [Follower] entering Follower state (Leader: "")
2021/04/14 11:21:05 [INFO] raft: aborting pipeline replication to peer {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}
2021/04/14 11:21:05 [INFO] consul: cluster leadership lost
2021/04/14 11:21:05 [WARN] consul.coordinate: Batch update failed: leadership lost while committing log
2021/04/14 11:21:09 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:52114->10.242.70.73:8300: i/o timeout
2021/04/14 11:21:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:42626->10.242.5.10:8301: i/o timeout
2021/04/14 11:21:09 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:21:09 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:21:09 [WARN] raft: AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300} rejected, sending older logs (next: 530)
2021/04/14 11:21:09 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.70.73
2021/04/14 11:21:09 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:21:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:42676->10.242.5.10:8301: i/o timeout
2021/04/14 11:21:12 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:21:15 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/14 11:21:15 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/14 11:21:15 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 19
2021/04/14 11:21:15 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.70.77:48712
2021/04/14 11:21:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:57910->10.242.70.73:8301: i/o timeout
2021/04/14 11:21:17 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:21:18 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/14 11:21:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:42792->10.242.5.10:8301: i/o timeout
2021/04/14 11:21:19 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:21:20 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:21:20 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 20
2021/04/14 11:21:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:57954->10.242.70.73:8301: i/o timeout
2021/04/14 11:21:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:21:23 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/04/14 11:21:23 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:21:23 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:21:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:42890->10.242.5.10:8301: i/o timeout
2021/04/14 11:21:25 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:21:25 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:52124->10.242.70.73:8300: i/o timeout
2021/04/14 11:21:25 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:60322->10.242.5.10:8300: i/o timeout
2021/04/14 11:21:25 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/04/14 11:21:25 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.70.73
2021/04/14 11:21:25 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:21:25 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:21:25 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.5.10:51690
2021/04/14 11:21:25 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:54544
2021/04/14 11:21:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:44396->10.242.5.10:8302: i/o timeout
2021/04/14 11:21:28 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:21:29 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.70.77:48712
2021/04/14 11:21:30 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:21:30 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 21
2021/04/14 11:21:30 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:21:30 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:21:30 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:60706->10.242.5.10:8300: i/o timeout
2021/04/14 11:21:30 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:52512->10.242.70.73:8300: i/o timeout
2021/04/14 11:21:31 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.70.73
2021/04/14 11:21:31 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:21:32 [INFO] raft: Election won. Tally: 2
2021/04/14 11:21:32 [INFO] raft: Node at 10.242.228.137:8300 [Leader] entering Leader state
2021/04/14 11:21:32 [INFO] raft: Added peer 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, starting replication
2021/04/14 11:21:32 [INFO] raft: Added peer 590cff07-d5aa-e855-db2c-f428e810df3b, starting replication
2021/04/14 11:21:32 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:21:32 [INFO] consul: cluster leadership acquired
2021/04/14 11:21:32 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/14 11:21:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:58202->10.242.70.73:8301: i/o timeout
2021/04/14 11:21:33 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:21:33 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:21:35 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 2.502094731s
2021/04/14 11:21:35 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 2.502082968s
2021/04/14 11:21:35 [WARN] raft: Failed to contact quorum of nodes, stepping down
2021/04/14 11:21:35 [INFO] raft: Node at 10.242.228.137:8300 [Follower] entering Follower state (Leader: "")
2021/04/14 11:21:35 [ERR] consul: failed to wait for barrier: leadership lost while committing log
2021/04/14 11:21:35 [INFO] consul: cluster leadership lost
2021/04/14 11:21:35 [WARN] consul.coordinate: Batch update failed: node is not the leader
2021/04/14 11:21:36 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/14 11:21:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:58316->10.242.70.73:8301: i/o timeout
2021/04/14 11:21:39 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:21:40 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:60872->10.242.5.10:8300: i/o timeout
2021/04/14 11:21:41 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/14 11:21:41 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 22
2021/04/14 11:21:41 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:21:41 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:21:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:58370->10.242.70.73:8301: i/o timeout
2021/04/14 11:21:42 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:21:42 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:21:42 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:52678->10.242.70.73:8300: i/o timeout
2021/04/14 11:21:42 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:60922->10.242.5.10:8300: i/o timeout
2021/04/14 11:21:43 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:21:43 [INFO] raft: Election won. Tally: 2
2021/04/14 11:21:43 [INFO] raft: Node at 10.242.228.137:8300 [Leader] entering Leader state
2021/04/14 11:21:43 [INFO] raft: Added peer 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, starting replication
2021/04/14 11:21:43 [INFO] raft: Added peer 590cff07-d5aa-e855-db2c-f428e810df3b, starting replication
2021/04/14 11:21:43 [INFO] consul: cluster leadership acquired
2021/04/14 11:21:43 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:21:43 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/14 11:21:43 [WARN] raft: AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300} rejected, sending older logs (next: 536)
2021/04/14 11:21:43 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:21:43 [INFO] raft: pipelining replication to peer {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}
2021/04/14 11:21:43 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:21:43 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/14 11:21:43 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/14 11:21:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:58474->10.242.70.73:8301: i/o timeout
2021/04/14 11:21:44 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:21:44 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:21:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:58510->10.242.70.73:8301: i/o timeout
2021/04/14 11:21:46 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:21:46 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/04/14 11:21:46 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 2.504876424s
2021/04/14 11:21:46 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/04/14 11:21:46 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.70.73
2021/04/14 11:21:46 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:21:46 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/14 11:21:46 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 2.500329175s
2021/04/14 11:21:46 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 2.665453089s
2021/04/14 11:21:46 [WARN] raft: Failed to contact quorum of nodes, stepping down
2021/04/14 11:21:46 [INFO] raft: Node at 10.242.228.137:8300 [Follower] entering Follower state (Leader: "")
2021/04/14 11:21:46 [INFO] raft: aborting pipeline replication to peer {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}
2021/04/14 11:21:46 [ERR] consul: failed to reconcile member: {dev-consul-server-2 10.242.70.73 8301 map[role:consul port:8300 vsn:2 build:1.0.6:9a494b5f expect:3 dc:dc1 raft_vsn:3 wan_join_port:8302 vsn_min:2 segment: id:590cff07-d5aa-e855-db2c-f428e810df3b vsn_max:3] failed 1 5 2 2 5 4}: leadership lost while committing log
2021/04/14 11:21:46 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:21:46 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:21:46 [INFO] consul: cluster leadership lost
2021/04/14 11:21:47 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:21:49 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:21:49 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.70.73:52692
2021/04/14 11:21:49 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:21:49 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/14 11:21:49 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: read tcp 10.242.228.137:52746->10.242.70.73:8300: i/o timeout
2021/04/14 11:21:50 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:21:51 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:52888->10.242.70.73:8300: i/o timeout
2021/04/14 11:21:52 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:52976->10.242.70.73:8300: i/o timeout
2021/04/14 11:21:53 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:53004->10.242.70.73:8300: i/o timeout
2021/04/14 11:21:53 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:21:54 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/14 11:21:54 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 23
2021/04/14 11:21:54 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:21:54 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:21:54 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: read tcp 10.242.228.137:53018->10.242.70.73:8300: i/o timeout
2021/04/14 11:21:55 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:21:55 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.70.73
2021/04/14 11:21:55 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:21:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:45074->10.242.5.10:8302: i/o timeout
2021/04/14 11:21:56 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:21:57 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:21:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:58794->10.242.70.73:8301: i/o timeout
2021/04/14 11:21:59 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:21:59 [WARN] raft: AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300} rejected, sending older logs (next: 530)
2021/04/14 11:21:59 [INFO] raft: Election won. Tally: 2
2021/04/14 11:21:59 [INFO] raft: Node at 10.242.228.137:8300 [Leader] entering Leader state
2021/04/14 11:21:59 [INFO] raft: Added peer 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, starting replication
2021/04/14 11:21:59 [INFO] raft: Added peer 590cff07-d5aa-e855-db2c-f428e810df3b, starting replication
2021/04/14 11:21:59 [INFO] consul: cluster leadership acquired
2021/04/14 11:21:59 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:21:59 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/14 11:21:59 [WARN] raft: AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300} rejected, sending older logs (next: 530)
2021/04/14 11:21:59 [INFO] raft: pipelining replication to peer {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}
2021/04/14 11:21:59 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/14 11:22:00 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:22:02 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:22:02 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:22:02 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/14 11:22:02 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 2.504027334s
2021/04/14 11:22:02 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:22:02 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:22:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:04 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:32850->10.242.5.10:8300: i/o timeout
2021/04/14 11:22:04 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 4.972326471s
2021/04/14 11:22:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:43794->10.242.5.10:8301: i/o timeout
2021/04/14 11:22:06 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:22:07 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 7.458309777s
2021/04/14 11:22:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:09 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/14 11:22:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:45384->10.242.5.10:8302: i/o timeout
2021/04/14 11:22:09 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:22:09 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:33310->10.242.5.10:8300: i/o timeout
2021/04/14 11:22:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:10 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:33324->10.242.5.10:8300: i/o timeout
2021/04/14 11:22:10 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:22:10 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:22:10 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/14 11:22:11 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:22:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:19 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:22:19 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/14 11:22:19 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:33504->10.242.5.10:8300: i/o timeout
2021/04/14 11:22:19 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:22:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:22:20 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:22:21 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:33532->10.242.5.10:8300: i/o timeout
2021/04/14 11:22:21 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:22:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:59290->10.242.70.73:8301: i/o timeout
2021/04/14 11:22:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:45804->10.242.5.10:8302: i/o timeout
2021/04/14 11:22:23 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:22:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:27 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:22:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:28 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:22:29 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:33730->10.242.5.10:8300: i/o timeout
2021/04/14 11:22:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:22:29 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:22:31 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:33782->10.242.5.10:8300: i/o timeout
2021/04/14 11:22:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:32 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:22:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:38 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/04/14 11:22:39 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:22:39 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 11:22:39 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 11:22:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:39 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:33966->10.242.5.10:8300: i/o timeout
2021/04/14 11:22:40 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:22:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:42 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:34036->10.242.5.10:8300: i/o timeout
2021/04/14 11:22:43 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.5.10:8302
2021/04/14 11:22:43 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:22:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:46 [ERR] memberlist: failed to receive: read tcp 10.242.228.137:8301->10.242.5.10:52708: i/o timeout from=10.242.5.10:52708
2021/04/14 11:22:46 [ERR] memberlist: Failed to send error: write tcp 10.242.228.137:8301->10.242.5.10:52708: i/o timeout from=10.242.5.10:52708
2021/04/14 11:22:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:50 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:34150->10.242.5.10:8300: i/o timeout
2021/04/14 11:22:50 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:22:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:53 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:34268->10.242.5.10:8300: i/o timeout
2021/04/14 11:22:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:54 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:22:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:22:59 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:22:59 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/14 11:23:00 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:34384->10.242.5.10:8300: i/o timeout
2021/04/14 11:23:00 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:23:02 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:23:02 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:23:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:23:04 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:34476->10.242.5.10:8300: i/o timeout
2021/04/14 11:23:05 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:23:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:23:06 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:23:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:23:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:23:10 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:34608->10.242.5.10:8300: i/o timeout
2021/04/14 11:23:10 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:23:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:23:12 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 11:23:12 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 11:23:12 [WARN] raft: Rejecting vote request from 10.242.5.10:8300 since we have a leader: 10.242.228.137:8300
2021/04/14 11:23:12 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:23:12 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:23:12 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/14 11:23:12 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/04/14 11:23:12 [ERR] raft: peer {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300} has newer term, stopping replication
2021/04/14 11:23:12 [INFO] raft: Node at 10.242.228.137:8300 [Follower] entering Follower state (Leader: "")
2021/04/14 11:23:12 [INFO] raft: aborting pipeline replication to peer {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}
2021/04/14 11:23:12 [INFO] consul: cluster leadership lost
2021/04/14 11:23:18 [INFO] consul: New leader elected: dev-consul-server-2
2021/04/14 11:23:18 [DEBUG] raft-net: 10.242.228.137:8300 accepted connection from: 10.242.70.73:54954
2021/04/14 11:23:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:60758->10.242.70.73:8301: i/o timeout
2021/04/14 11:24:18 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/04/14 11:24:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:33562->10.242.70.73:8301: i/o timeout
2021/04/14 11:24:21 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:24:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:33584->10.242.70.73:8301: i/o timeout
2021/04/14 11:24:23 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:24:23 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/14 11:24:23 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.70.73
2021/04/14 11:24:23 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:24:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:46746->10.242.5.10:8301: i/o timeout
2021/04/14 11:24:25 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:24:25 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/14 11:24:25 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:24:25 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:24:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:48458->10.242.5.10:8302: i/o timeout
2021/04/14 11:24:28 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:24:38 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:24:38 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.5.10:53558
2021/04/14 11:24:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:48556->10.242.5.10:8302: i/o timeout
2021/04/14 11:24:38 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:24:40 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:24:40 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.70.73:53454
2021/04/14 11:24:41 [WARN] raft: Heartbeat timeout from "10.242.70.73:8300" reached, starting election
2021/04/14 11:24:41 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 25
2021/04/14 11:24:41 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:24:41 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:24:42 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:24:42 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.70.73:8301
2021/04/14 11:24:42 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:24:42 [ERR] http: Request PUT /v1/txn, error: rpc error making call: EOF from=10.242.198.137:57342
2021/04/14 11:24:43 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.70.73
2021/04/14 11:24:43 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:24:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:34140->10.242.70.73:8301: i/o timeout
2021/04/14 11:24:47 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:24:47 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:24:47 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 26
2021/04/14 11:24:47 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:24:49 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:56178
2021/04/14 11:24:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:34178->10.242.70.73:8301: i/o timeout
2021/04/14 11:24:50 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:24:50 [INFO] raft: Node at 10.242.228.137:8300 [Follower] entering Follower state (Leader: "")
2021/04/14 11:24:51 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:34810->10.242.5.10:8300: i/o timeout
2021/04/14 11:24:51 [WARN] raft: Failed to get previous log: 575 log not found (last: 574)
2021/04/14 11:24:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:34260->10.242.70.73:8301: i/o timeout
2021/04/14 11:24:53 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:24:53 [INFO] consul: New leader elected: dev-consul-server-2
2021/04/14 11:24:53 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/14 11:24:53 [DEBUG] raft-net: 10.242.228.137:8300 accepted connection from: 10.242.70.73:56162
2021/04/14 11:24:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:48972->10.242.5.10:8302: i/o timeout
2021/04/14 11:24:53 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:24:57 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:34712->10.242.5.10:8300: i/o timeout
2021/04/14 11:24:58 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:24:58 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 11:24:58 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 11:24:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:34404->10.242.70.73:8301: i/o timeout
2021/04/14 11:24:59 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:25:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:34446->10.242.70.73:8301: i/o timeout
2021/04/14 11:25:01 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:25:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:34498->10.242.70.73:8301: i/o timeout
2021/04/14 11:25:03 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:25:03 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:25:03 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.70.73
2021/04/14 11:25:03 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:25:04 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.70.73
2021/04/14 11:25:04 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:25:05 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/14 11:25:05 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:25:05 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:25:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:47668->10.242.5.10:8301: i/o timeout
2021/04/14 11:25:07 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:25:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:47700->10.242.5.10:8301: i/o timeout
2021/04/14 11:25:10 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:25:12 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.5.10:8302
2021/04/14 11:25:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:34650->10.242.70.73:8301: i/o timeout
2021/04/14 11:25:13 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:25:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:37900->10.242.70.73:8302: i/o timeout
2021/04/14 11:25:13 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/14 11:25:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:47850->10.242.5.10:8301: i/o timeout
2021/04/14 11:25:16 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:25:16 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:25:16 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:25:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:34904->10.242.70.73:8301: i/o timeout
2021/04/14 11:25:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:25:24 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/14 11:25:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:35170->10.242.70.73:8301: i/o timeout
2021/04/14 11:25:33 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:25:36 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/14 11:25:36 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/14 11:25:43 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:25:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:35678->10.242.70.73:8301: i/o timeout
2021/04/14 11:26:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:35866->10.242.70.73:8301: i/o timeout
2021/04/14 11:26:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:36188->10.242.70.73:8301: i/o timeout
2021/04/14 11:26:18 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:26:19 [WARN] raft: Rejecting vote request from 10.242.5.10:8300 since we have a leader: 10.242.70.73:8300
2021/04/14 11:26:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:36254->10.242.70.73:8301: i/o timeout
2021/04/14 11:26:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:26:22 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.70.73
2021/04/14 11:26:22 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:26:23 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.70.73
2021/04/14 11:26:23 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:26:30 [ERR] agent: failed to sync remote state: rpc error making call: No cluster leader
2021/04/14 11:26:30 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:26:30 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:26:30 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 11:26:30 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 11:26:32 [INFO] consul: New leader elected: dev-consul-server-2
2021/04/14 11:26:32 [DEBUG] raft-net: 10.242.228.137:8300 accepted connection from: 10.242.70.73:57576
2021/04/14 11:26:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:49650->10.242.5.10:8301: i/o timeout
2021/04/14 11:26:36 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:26:36 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/14 11:26:36 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:26:36 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:26:53 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:26:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:51624->10.242.5.10:8302: i/o timeout
2021/04/14 11:26:53 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:26:58 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:26:58 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.5.10:55396
2021/04/14 11:26:59 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:26:59 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:27:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:50242->10.242.5.10:8301: i/o timeout
2021/04/14 11:27:03 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:27:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:37266->10.242.70.73:8301: i/o timeout
2021/04/14 11:27:08 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:27:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:50610->10.242.5.10:8301: i/o timeout
2021/04/14 11:27:17 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:27:19 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/14 11:27:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:37580->10.242.70.73:8301: i/o timeout
2021/04/14 11:27:20 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:27:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:50712->10.242.5.10:8301: i/o timeout
2021/04/14 11:27:22 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:27:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:40830->10.242.70.73:8302: i/o timeout
2021/04/14 11:27:23 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/14 11:27:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:37648->10.242.70.73:8301: i/o timeout
2021/04/14 11:27:24 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:27:25 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/14 11:27:25 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.70.73
2021/04/14 11:27:25 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:27:26 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/04/14 11:27:26 [WARN] raft: Heartbeat timeout from "10.242.70.73:8300" reached, starting election
2021/04/14 11:27:26 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 29
2021/04/14 11:27:26 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:27:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:50836->10.242.5.10:8301: i/o timeout
2021/04/14 11:27:27 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:27:28 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/04/14 11:27:28 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:27:28 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:27:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:50912->10.242.5.10:8301: i/o timeout
2021/04/14 11:27:29 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:27:29 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:27:30 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.70.73
2021/04/14 11:27:30 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:27:31 [INFO] raft: Duplicate RequestVote for same term: 29
2021/04/14 11:27:33 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:27:33 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: rpc error making call: EOF from=10.242.198.137:60056
2021/04/14 11:27:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:52504->10.242.5.10:8302: i/o timeout
2021/04/14 11:27:33 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:27:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:37920->10.242.70.73:8301: i/o timeout
2021/04/14 11:27:34 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:27:35 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:27:35 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 30
2021/04/14 11:27:35 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:27:35 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:27:35 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:27:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:37960->10.242.70.73:8301: i/o timeout
2021/04/14 11:27:36 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:27:36 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:53210->10.242.70.73:8300: i/o timeout
2021/04/14 11:27:36 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:40450->10.242.5.10:8300: i/o timeout
2021/04/14 11:27:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:51070->10.242.5.10:8301: i/o timeout
2021/04/14 11:27:39 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:27:40 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:27:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:38026->10.242.70.73:8301: i/o timeout
2021/04/14 11:27:42 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:27:44 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/04/14 11:27:44 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.70.73
2021/04/14 11:27:44 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:27:44 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:27:44 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 31
2021/04/14 11:27:44 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:27:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:38194->10.242.70.73:8301: i/o timeout
2021/04/14 11:27:45 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:27:45 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:53122->10.242.70.73:8300: i/o timeout
2021/04/14 11:27:45 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:40668->10.242.5.10:8300: i/o timeout
2021/04/14 11:27:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:51320->10.242.5.10:8301: i/o timeout
2021/04/14 11:27:47 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:27:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:41186->10.242.70.73:8302: i/o timeout
2021/04/14 11:27:48 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/14 11:27:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:51350->10.242.5.10:8301: i/o timeout
2021/04/14 11:27:49 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:27:49 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:60264
2021/04/14 11:27:51 [INFO] raft: Election won. Tally: 2
2021/04/14 11:27:51 [INFO] raft: Node at 10.242.228.137:8300 [Leader] entering Leader state
2021/04/14 11:27:51 [INFO] raft: Added peer 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, starting replication
2021/04/14 11:27:51 [INFO] raft: Added peer 590cff07-d5aa-e855-db2c-f428e810df3b, starting replication
2021/04/14 11:27:51 [INFO] consul: cluster leadership acquired
2021/04/14 11:27:51 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:27:51 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/14 11:27:51 [INFO] raft: pipelining replication to peer {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}
2021/04/14 11:27:52 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/14 11:27:52 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:27:53 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:27:53 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.70.73
2021/04/14 11:27:53 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/14 11:27:54 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 2.503929109s
2021/04/14 11:27:54 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:60958->10.242.70.73:8300: i/o timeout
2021/04/14 11:27:55 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:27:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:27:56 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 4.965927383s
2021/04/14 11:27:57 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:27:57 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.70.73:55960
2021/04/14 11:27:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:27:59 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 7.463951019s
2021/04/14 11:28:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:01 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:32864->10.242.70.73:8300: i/o timeout
2021/04/14 11:28:02 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:28:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:02 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: read tcp 10.242.228.137:32878->10.242.70.73:8300: i/o timeout
2021/04/14 11:28:03 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:28:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:41656->10.242.70.73:8302: i/o timeout
2021/04/14 11:28:08 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/14 11:28:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:12 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:33088->10.242.70.73:8300: i/o timeout
2021/04/14 11:28:12 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:28:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:13 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: read tcp 10.242.228.137:33128->10.242.70.73:8300: i/o timeout
2021/04/14 11:28:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:14 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:28:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:22 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:33290->10.242.70.73:8300: i/o timeout
2021/04/14 11:28:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:22 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:28:23 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:28:23 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/14 11:28:24 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: read tcp 10.242.228.137:33364->10.242.70.73:8300: i/o timeout
2021/04/14 11:28:24 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:28:25 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:28:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:32 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:33494->10.242.70.73:8300: i/o timeout
2021/04/14 11:28:32 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:28:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:34 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: read tcp 10.242.228.137:33554->10.242.70.73:8300: i/o timeout
2021/04/14 11:28:35 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:28:35 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:28:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:42 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:33716->10.242.70.73:8300: i/o timeout
2021/04/14 11:28:42 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:28:42 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.70.73:8302
2021/04/14 11:28:43 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:28:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:28:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:45 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: read tcp 10.242.228.137:33782->10.242.70.73:8300: i/o timeout
2021/04/14 11:28:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:46 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:28:47 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:28:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:51 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:28:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:52 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:33916->10.242.70.73:8300: i/o timeout
2021/04/14 11:28:52 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:28:53 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:28:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:28:56 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: read tcp 10.242.228.137:34036->10.242.70.73:8300: i/o timeout
2021/04/14 11:28:57 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:28:57 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:28:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:29:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:29:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:52886->10.242.5.10:8301: i/o timeout
2021/04/14 11:29:01 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:29:01 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:29:01 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/14 11:29:01 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 2.500995722s
2021/04/14 11:29:01 [WARN] raft: Failed to contact quorum of nodes, stepping down
2021/04/14 11:29:01 [INFO] raft: aborting pipeline replication to peer {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}
2021/04/14 11:29:01 [INFO] raft: Node at 10.242.228.137:8300 [Follower] entering Follower state (Leader: "")
2021/04/14 11:29:01 [INFO] consul: cluster leadership lost
2021/04/14 11:29:02 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:34140->10.242.70.73:8300: i/o timeout
2021/04/14 11:29:02 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:29:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:52968->10.242.5.10:8301: i/o timeout
2021/04/14 11:29:04 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:29:04 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:29:04 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/14 11:29:05 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:29:05 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:29:05 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:29:06 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:29:06 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:29:07 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: read tcp 10.242.228.137:34254->10.242.70.73:8300: i/o timeout
2021/04/14 11:29:07 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/14 11:29:08 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:29:08 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/14 11:29:08 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 32
2021/04/14 11:29:08 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:29:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:53092->10.242.5.10:8301: i/o timeout
2021/04/14 11:29:12 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:34374->10.242.70.73:8300: i/o timeout
2021/04/14 11:29:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:53148->10.242.5.10:8301: i/o timeout
2021/04/14 11:29:13 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:29:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:53214->10.242.5.10:8301: i/o timeout
2021/04/14 11:29:16 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:29:17 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:29:17 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:29:17 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:29:18 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: read tcp 10.242.228.137:34462->10.242.70.73:8300: i/o timeout
2021/04/14 11:29:18 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:29:18 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 33
2021/04/14 11:29:18 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:29:18 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:29:18 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:41074->10.242.5.10:8300: i/o timeout
2021/04/14 11:29:18 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:34470->10.242.70.73:8300: i/o timeout
2021/04/14 11:29:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:54872->10.242.5.10:8302: i/o timeout
2021/04/14 11:29:18 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:29:19 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:33948
2021/04/14 11:29:25 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:29:26 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:29:26 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 34
2021/04/14 11:29:26 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:29:26 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:29:28 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:34686->10.242.70.73:8300: i/o timeout
2021/04/14 11:29:28 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:42880->10.242.5.10:8300: i/o timeout
2021/04/14 11:29:32 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:29:32 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.5.10:56368
2021/04/14 11:29:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:55088->10.242.5.10:8302: i/o timeout
2021/04/14 11:29:33 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:29:34 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:29:35 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:29:35 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 35
2021/04/14 11:29:35 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:29:35 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:29:36 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:43044->10.242.5.10:8300: i/o timeout
2021/04/14 11:29:36 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:34854->10.242.70.73:8300: i/o timeout
2021/04/14 11:29:39 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.70.73:8301
2021/04/14 11:29:39 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:29:42 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:29:42 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 36
2021/04/14 11:29:43 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:29:43 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:29:45 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:43256->10.242.5.10:8300: i/o timeout
2021/04/14 11:29:45 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:35062->10.242.70.73:8300: i/o timeout
2021/04/14 11:29:47 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:29:47 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:29:48 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/04/14 11:29:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:53922->10.242.5.10:8301: i/o timeout
2021/04/14 11:29:49 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:29:49 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:34590
2021/04/14 11:29:50 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:29:50 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 37
2021/04/14 11:29:50 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:29:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:53972->10.242.5.10:8301: i/o timeout
2021/04/14 11:29:52 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:29:52 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.70.73
2021/04/14 11:29:52 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/14 11:29:52 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.70.73
2021/04/14 11:29:52 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:29:52 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:29:53 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:29:53 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:43442->10.242.5.10:8300: i/o timeout
2021/04/14 11:29:53 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:29:53 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:35248->10.242.70.73:8300: i/o timeout
2021/04/14 11:29:53 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:29:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:54010->10.242.5.10:8301: i/o timeout
2021/04/14 11:29:54 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:29:54 [INFO] raft: Election won. Tally: 2
2021/04/14 11:29:54 [INFO] raft: Node at 10.242.228.137:8300 [Leader] entering Leader state
2021/04/14 11:29:54 [INFO] raft: Added peer 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, starting replication
2021/04/14 11:29:54 [INFO] raft: Added peer 590cff07-d5aa-e855-db2c-f428e810df3b, starting replication
2021/04/14 11:29:54 [INFO] consul: cluster leadership acquired
2021/04/14 11:29:54 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/14 11:29:54 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:29:54 [WARN] raft: AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300} rejected, sending older logs (next: 606)
2021/04/14 11:29:54 [INFO] raft: pipelining replication to peer {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}
2021/04/14 11:29:54 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/14 11:29:54 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/14 11:29:55 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:29:56 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: stream closed
2021/04/14 11:29:57 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 2.50216541s
2021/04/14 11:29:57 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:29:57 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:29:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:41068->10.242.70.73:8301: i/o timeout
2021/04/14 11:29:59 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:29:59 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 4.961454778s
2021/04/14 11:29:59 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:29:59 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:30:00 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:43580->10.242.5.10:8300: i/o timeout
2021/04/14 11:30:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:02 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 7.128895863s
2021/04/14 11:30:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:55866->10.242.5.10:8302: i/o timeout
2021/04/14 11:30:03 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:30:04 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:43672->10.242.5.10:8300: i/o timeout
2021/04/14 11:30:04 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:30:04 [ERR] memberlist: failed to receive: read tcp 10.242.228.137:8301->10.242.5.10:57896: i/o timeout from=10.242.5.10:57896
2021/04/14 11:30:04 [ERR] memberlist: Failed to send error: write tcp 10.242.228.137:8301->10.242.5.10:57896: i/o timeout from=10.242.5.10:57896
2021/04/14 11:30:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:05 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:43688->10.242.5.10:8300: i/o timeout
2021/04/14 11:30:06 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:30:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:13 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:30:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:55960->10.242.5.10:8302: i/o timeout
2021/04/14 11:30:14 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:30:14 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:43908->10.242.5.10:8300: i/o timeout
2021/04/14 11:30:14 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:30:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:41442->10.242.70.73:8301: i/o timeout
2021/04/14 11:30:16 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:30:16 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:43936->10.242.5.10:8300: i/o timeout
2021/04/14 11:30:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:17 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:30:17 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:30:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:41486->10.242.70.73:8301: i/o timeout
2021/04/14 11:30:18 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:30:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:30:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:41514->10.242.70.73:8301: i/o timeout
2021/04/14 11:30:20 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:30:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:30:21 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/14 11:30:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:23 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:30:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:56300->10.242.5.10:8302: i/o timeout
2021/04/14 11:30:23 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:30:24 [ERR] memberlist: failed to receive: read tcp 10.242.228.137:8302->10.242.5.10:46216: i/o timeout from=10.242.5.10:46216
2021/04/14 11:30:24 [ERR] memberlist: Failed to send error: write tcp 10.242.228.137:8302->10.242.5.10:46216: i/o timeout from=10.242.5.10:46216
2021/04/14 11:30:24 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:44146->10.242.5.10:8300: i/o timeout
2021/04/14 11:30:24 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:30:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:30:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:41626->10.242.70.73:8301: i/o timeout
2021/04/14 11:30:25 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:30:25 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 2.500296825s
2021/04/14 11:30:25 [WARN] raft: Failed to contact quorum of nodes, stepping down
2021/04/14 11:30:25 [INFO] raft: aborting pipeline replication to peer {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}
2021/04/14 11:30:25 [INFO] raft: Node at 10.242.228.137:8300 [Follower] entering Follower state (Leader: "")
2021/04/14 11:30:25 [WARN] consul.coordinate: Batch update failed: leadership lost while committing log
2021/04/14 11:30:25 [INFO] consul: cluster leadership lost
2021/04/14 11:30:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:41658->10.242.70.73:8301: i/o timeout
2021/04/14 11:30:27 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:30:27 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:44198->10.242.5.10:8300: i/o timeout
2021/04/14 11:30:28 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:30:32 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/14 11:30:32 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 38
2021/04/14 11:30:32 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:30:32 [INFO] raft: Election won. Tally: 2
2021/04/14 11:30:32 [INFO] raft: Node at 10.242.228.137:8300 [Leader] entering Leader state
2021/04/14 11:30:32 [INFO] consul: cluster leadership acquired
2021/04/14 11:30:32 [INFO] raft: Added peer 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, starting replication
2021/04/14 11:30:32 [INFO] raft: Added peer 590cff07-d5aa-e855-db2c-f428e810df3b, starting replication
2021/04/14 11:30:32 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:30:32 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/14 11:30:32 [WARN] raft: AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300} rejected, sending older logs (next: 628)
2021/04/14 11:30:33 [INFO] raft: pipelining replication to peer {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}
2021/04/14 11:30:33 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:30:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:56540->10.242.5.10:8302: i/o timeout
2021/04/14 11:30:33 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:30:33 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:30:33 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 11:30:33 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 11:30:34 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:44348->10.242.5.10:8300: i/o timeout
2021/04/14 11:30:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:35 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 2.506253453s
2021/04/14 11:30:36 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:30:36 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/14 11:30:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:41900->10.242.70.73:8301: i/o timeout
2021/04/14 11:30:37 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:30:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:30:37 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 4.581377427s
2021/04/14 11:30:38 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:44446->10.242.5.10:8300: i/o timeout
2021/04/14 11:30:38 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:30:38 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/14 11:30:39 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 6.998284192s
2021/04/14 11:30:40 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:30:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:42 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.5.10:8302
2021/04/14 11:30:42 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:44546->10.242.5.10:8300: i/o timeout
2021/04/14 11:30:42 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:44548->10.242.5.10:8300: i/o timeout
2021/04/14 11:30:42 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:30:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:43 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:44568->10.242.5.10:8300: i/o timeout
2021/04/14 11:30:44 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:30:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:46 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:30:46 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:30:46 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/14 11:30:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:55304->10.242.5.10:8301: i/o timeout
2021/04/14 11:30:51 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:30:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:52 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:44784->10.242.5.10:8300: i/o timeout
2021/04/14 11:30:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:53 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:30:53 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:30:53 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/14 11:30:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:55366->10.242.5.10:8301: i/o timeout
2021/04/14 11:30:54 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:30:54 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:44830->10.242.5.10:8300: i/o timeout
2021/04/14 11:30:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:55 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:30:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:30:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:31:00 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:31:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:31:02 [WARN] raft: Rejecting vote request from 10.242.5.10:8300 since we have a leader: 10.242.228.137:8300
2021/04/14 11:31:03 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:44974->10.242.5.10:8300: i/o timeout
2021/04/14 11:31:03 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:31:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:31:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:31:05 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:45018->10.242.5.10:8300: i/o timeout
2021/04/14 11:31:05 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:31:06 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/04/14 11:31:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:31:08 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 11:31:08 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 11:31:08 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:31:08 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:31:08 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/14 11:31:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:31:10 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:31:10 [WARN] raft: AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300} rejected, sending older logs (next: 618)
2021/04/14 11:31:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:31:11 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 2.500249317s
2021/04/14 11:31:11 [INFO] raft: pipelining replication to peer {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}
2021/04/14 11:31:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:42644->10.242.70.73:8301: i/o timeout
2021/04/14 11:31:12 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:31:12 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:31:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:31:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:31:13 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 4.691700024s
2021/04/14 11:31:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:42700->10.242.70.73:8301: i/o timeout
2021/04/14 11:31:15 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:31:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:31:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:31:26 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:31:26 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/14 11:31:48 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/14 11:31:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:43484->10.242.70.73:8301: i/o timeout
2021/04/14 11:31:50 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:31:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:31:50 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/14 11:31:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:31:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:43548->10.242.70.73:8301: i/o timeout
2021/04/14 11:31:53 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:31:53 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.70.73
2021/04/14 11:31:53 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:31:53 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/14 11:31:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:31:53 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 2.500198498s
2021/04/14 11:31:55 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:31:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:31:56 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/14 11:31:56 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:31:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:31:57 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 2.500248241s
2021/04/14 11:31:58 [ERR] raft: Failed to pipeline AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: write tcp 10.242.228.137:35498->10.242.70.73:8300: use of closed network connection
2021/04/14 11:31:58 [INFO] raft: aborting pipeline replication to peer {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}
2021/04/14 11:31:58 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:31:58 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.70.73
2021/04/14 11:31:58 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:31:58 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/14 11:31:58 [INFO] raft: pipelining replication to peer {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}
2021/04/14 11:32:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:43832->10.242.70.73:8301: i/o timeout
2021/04/14 11:32:07 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:32:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:32:08 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/14 11:32:08 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 2.500367676s
2021/04/14 11:32:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:47038->10.242.70.73:8302: i/o timeout
2021/04/14 11:32:08 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/14 11:32:09 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/14 11:32:09 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.70.73
2021/04/14 11:32:09 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:32:09 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/14 11:32:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:32:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:43866->10.242.70.73:8301: i/o timeout
2021/04/14 11:32:10 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:32:10 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 4.995381373s
2021/04/14 11:32:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:32:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:32:13 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 7.414555409s
2021/04/14 11:32:14 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: read tcp 10.242.228.137:36370->10.242.70.73:8300: i/o timeout
2021/04/14 11:32:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:32:15 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:32:15 [INFO] raft: aborting pipeline replication to peer {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}
2021/04/14 11:32:16 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:32:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:32:18 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 2.500317218s
2021/04/14 11:32:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:32:19 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.70.73
2021/04/14 11:32:19 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:32:19 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/14 11:32:20 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 4.928916492s
2021/04/14 11:32:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:32:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:44192->10.242.70.73:8301: i/o timeout
2021/04/14 11:32:23 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:32:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:32:23 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 7.33872967s
2021/04/14 11:32:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:47374->10.242.70.73:8302: i/o timeout
2021/04/14 11:32:23 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/14 11:32:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:32:25 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/14 11:32:25 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.70.73
2021/04/14 11:32:25 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:32:25 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/14 11:32:25 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: read tcp 10.242.228.137:38562->10.242.70.73:8300: i/o timeout
2021/04/14 11:32:25 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:32:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:44240->10.242.70.73:8301: i/o timeout
2021/04/14 11:32:26 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:32:26 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:38572->10.242.70.73:8300: i/o timeout
2021/04/14 11:32:26 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:32:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:32:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:32:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:32:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:32:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:47586->10.242.70.73:8302: i/o timeout
2021/04/14 11:32:34 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/14 11:32:34 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:32:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:57540->10.242.5.10:8301: i/o timeout
2021/04/14 11:32:35 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:32:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:32:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:32:35 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: read tcp 10.242.228.137:38768->10.242.70.73:8300: i/o timeout
2021/04/14 11:32:36 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:38770->10.242.70.73:8300: i/o timeout
2021/04/14 11:32:36 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:32:36 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:32:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:32:38 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:32:38 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.70.73
2021/04/14 11:32:38 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/14 11:32:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:32:40 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:32:40 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/14 11:32:42 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:32:42 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:32:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:57726->10.242.5.10:8301: i/o timeout
2021/04/14 11:32:43 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:32:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:32:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:32:43 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.70.73
2021/04/14 11:32:43 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/14 11:32:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:47800->10.242.70.73:8302: i/o timeout
2021/04/14 11:32:43 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/14 11:32:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:57792->10.242.5.10:8301: i/o timeout
2021/04/14 11:32:45 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:32:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:32:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:32:46 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:38998->10.242.70.73:8300: i/o timeout
2021/04/14 11:32:46 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:32:46 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: read tcp 10.242.228.137:39012->10.242.70.73:8300: i/o timeout
2021/04/14 11:32:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:57810->10.242.5.10:8301: i/o timeout
2021/04/14 11:32:47 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:32:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:32:47 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:32:48 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:32:48 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 2.500237326s
2021/04/14 11:32:48 [WARN] raft: Failed to contact quorum of nodes, stepping down
2021/04/14 11:32:48 [INFO] raft: Node at 10.242.228.137:8300 [Follower] entering Follower state (Leader: "")
2021/04/14 11:32:48 [INFO] raft: aborting pipeline replication to peer {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}
2021/04/14 11:32:48 [INFO] consul: cluster leadership lost
2021/04/14 11:32:49 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.70.73:8301
2021/04/14 11:32:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:57898->10.242.5.10:8301: i/o timeout
2021/04/14 11:32:51 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:32:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:57912->10.242.5.10:8301: i/o timeout
2021/04/14 11:32:53 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:32:53 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/04/14 11:32:53 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:32:53 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:32:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:59474->10.242.5.10:8302: i/o timeout
2021/04/14 11:32:53 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:32:55 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/14 11:32:55 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 39
2021/04/14 11:32:55 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:32:55 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:32:56 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:39250->10.242.70.73:8300: i/o timeout
2021/04/14 11:32:56 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:32:56 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.5.10:58970
2021/04/14 11:32:57 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: read tcp 10.242.228.137:39280->10.242.70.73:8300: i/o timeout
2021/04/14 11:32:58 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:33:04 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:33:04 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 40
2021/04/14 11:33:04 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:33:04 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:33:05 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:45256->10.242.5.10:8300: i/o timeout
2021/04/14 11:33:05 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:39440->10.242.70.73:8300: i/o timeout
2021/04/14 11:33:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:48148->10.242.70.73:8302: i/o timeout
2021/04/14 11:33:08 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/14 11:33:10 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:33:10 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.70.73:59404
2021/04/14 11:33:12 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:33:12 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 41
2021/04/14 11:33:12 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:33:12 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:33:13 [ERR] memberlist: failed to receive: read tcp 10.242.228.137:8302->10.242.70.73:59932: i/o timeout from=10.242.70.73:59932
2021/04/14 11:33:13 [ERR] memberlist: Failed to send error: write tcp 10.242.228.137:8302->10.242.70.73:59932: i/o timeout from=10.242.70.73:59932
2021/04/14 11:33:14 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:47832->10.242.5.10:8300: i/o timeout
2021/04/14 11:33:14 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:39642->10.242.70.73:8300: i/o timeout
2021/04/14 11:33:15 [ERR] memberlist: Push/Pull with dev-consul-server-1.dc1 failed: read tcp 10.242.228.137:59448->10.242.5.10:8302: i/o timeout
2021/04/14 11:33:18 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:33:18 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 42
2021/04/14 11:33:18 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:33:18 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:33:20 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:33:20 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/14 11:33:21 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:33:21 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/14 11:33:22 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:47976->10.242.5.10:8300: i/o timeout
2021/04/14 11:33:22 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:39782->10.242.70.73:8300: i/o timeout
2021/04/14 11:33:23 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:33:23 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 11:33:23 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 11:33:28 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:33:28 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 43
2021/04/14 11:33:28 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:33:28 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:33:28 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:39958->10.242.70.73:8300: i/o timeout
2021/04/14 11:33:28 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:48152->10.242.5.10:8300: i/o timeout
2021/04/14 11:33:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:59980->10.242.5.10:8302: i/o timeout
2021/04/14 11:33:28 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:33:29 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.70.73:8301
2021/04/14 11:33:32 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:33:35 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:33:35 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 44
2021/04/14 11:33:35 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:33:35 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:33:38 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:40170->10.242.70.73:8300: i/o timeout
2021/04/14 11:33:38 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:48364->10.242.5.10:8300: i/o timeout
2021/04/14 11:33:38 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.5.10:8302
2021/04/14 11:33:38 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:33:38 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.70.73
2021/04/14 11:33:38 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/14 11:33:42 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:33:42 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 45
2021/04/14 11:33:42 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:33:42 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:33:45 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:40348->10.242.70.73:8300: i/o timeout
2021/04/14 11:33:45 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:48542->10.242.5.10:8300: i/o timeout
2021/04/14 11:33:48 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:33:48 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 46
2021/04/14 11:33:48 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:33:48 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:33:52 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:48696->10.242.5.10:8300: i/o timeout
2021/04/14 11:33:52 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:40502->10.242.70.73:8300: i/o timeout
2021/04/14 11:33:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:48924->10.242.70.73:8302: i/o timeout
2021/04/14 11:33:53 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/14 11:33:56 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:33:56 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 47
2021/04/14 11:33:56 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:33:56 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:33:58 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:48804->10.242.5.10:8300: i/o timeout
2021/04/14 11:33:58 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:40610->10.242.70.73:8300: i/o timeout
2021/04/14 11:34:04 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:34:04 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 48
2021/04/14 11:34:04 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:34:04 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:34:06 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:48986->10.242.5.10:8300: i/o timeout
2021/04/14 11:34:06 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:40796->10.242.70.73:8300: i/o timeout
2021/04/14 11:34:08 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:34:09 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.70.73:8301
2021/04/14 11:34:10 [ERR] http: Request GET /v1/catalog/service/holmes-rule-mgmt, error: No cluster leader from=10.242.70.77:57686
2021/04/14 11:34:11 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.70.73
2021/04/14 11:34:11 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:34:11 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/14 11:34:12 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:34:12 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 49
2021/04/14 11:34:12 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:34:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:46630->10.242.70.73:8301: i/o timeout
2021/04/14 11:34:13 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:34:14 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:40952->10.242.70.73:8300: i/o timeout
2021/04/14 11:34:14 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:49142->10.242.5.10:8300: i/o timeout
2021/04/14 11:34:14 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.70.73
2021/04/14 11:34:14 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/14 11:34:14 [INFO] raft: Election won. Tally: 2
2021/04/14 11:34:14 [INFO] raft: Node at 10.242.228.137:8300 [Leader] entering Leader state
2021/04/14 11:34:14 [INFO] raft: Added peer 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, starting replication
2021/04/14 11:34:14 [INFO] raft: Added peer 590cff07-d5aa-e855-db2c-f428e810df3b, starting replication
2021/04/14 11:34:14 [INFO] consul: cluster leadership acquired
2021/04/14 11:34:14 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:34:14 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/14 11:34:14 [INFO] raft: pipelining replication to peer {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}
2021/04/14 11:34:14 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/14 11:34:14 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/14 11:34:15 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:34:17 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 2.502544504s
2021/04/14 11:34:17 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:34:17 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:34:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:46752->10.242.70.73:8301: i/o timeout
2021/04/14 11:34:18 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:34:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:34:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:34:19 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 4.45425615s
2021/04/14 11:34:19 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 2.500356307s
2021/04/14 11:34:19 [WARN] raft: Failed to contact quorum of nodes, stepping down
2021/04/14 11:34:19 [INFO] raft: Node at 10.242.228.137:8300 [Follower] entering Follower state (Leader: "")
2021/04/14 11:34:19 [INFO] raft: aborting pipeline replication to peer {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}
2021/04/14 11:34:19 [INFO] consul: cluster leadership lost
2021/04/14 11:34:19 [WARN] consul.coordinate: Batch update failed: leadership lost while committing log
2021/04/14 11:34:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:46826->10.242.70.73:8301: i/o timeout
2021/04/14 11:34:21 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:34:22 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:49348->10.242.5.10:8300: i/o timeout
2021/04/14 11:34:24 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/14 11:34:24 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 50
2021/04/14 11:34:24 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:34:24 [INFO] raft: Election won. Tally: 2
2021/04/14 11:34:24 [INFO] raft: Node at 10.242.228.137:8300 [Leader] entering Leader state
2021/04/14 11:34:24 [INFO] consul: cluster leadership acquired
2021/04/14 11:34:24 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/14 11:34:24 [INFO] raft: Added peer 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, starting replication
2021/04/14 11:34:24 [INFO] raft: Added peer 590cff07-d5aa-e855-db2c-f428e810df3b, starting replication
2021/04/14 11:34:24 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:34:24 [WARN] raft: AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300} rejected, sending older logs (next: 666)
2021/04/14 11:34:24 [INFO] raft: pipelining replication to peer {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}
2021/04/14 11:34:24 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:49408->10.242.5.10:8300: i/o timeout
2021/04/14 11:34:24 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:34:25 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:49416->10.242.5.10:8300: i/o timeout
2021/04/14 11:34:25 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:34:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:46988->10.242.70.73:8301: i/o timeout
2021/04/14 11:34:26 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:34:26 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:34:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:34:27 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 2.504366444s
2021/04/14 11:34:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:34:29 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 5.001781827s
2021/04/14 11:34:29 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:34:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:34:31 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 7.087554199s
2021/04/14 11:34:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:34:34 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/14 11:34:34 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:49678->10.242.5.10:8300: i/o timeout
2021/04/14 11:34:34 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:49680->10.242.5.10:8300: i/o timeout
2021/04/14 11:34:34 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:34:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:34:34 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:49688->10.242.5.10:8300: i/o timeout
2021/04/14 11:34:35 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:49706->10.242.5.10:8300: i/o timeout
2021/04/14 11:34:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:47202->10.242.70.73:8301: i/o timeout
2021/04/14 11:34:36 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:34:36 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:34:36 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:49722->10.242.5.10:8300: i/o timeout
2021/04/14 11:34:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:34:36 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/14 11:34:38 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.5.10:8302
2021/04/14 11:34:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:34:40 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/14 11:34:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:34:41 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:34:41 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:34:41 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.70.73
2021/04/14 11:34:41 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:34:41 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/14 11:34:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:47360->10.242.70.73:8301: i/o timeout
2021/04/14 11:34:42 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:34:42 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:34:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:34:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:34:44 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:34:44 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:49904->10.242.5.10:8300: i/o timeout
2021/04/14 11:34:44 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:34:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:34:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:34:45 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:34:46 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:49962->10.242.5.10:8300: i/o timeout
2021/04/14 11:34:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:34:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:34:46 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 2.500314245s
2021/04/14 11:34:46 [WARN] raft: Failed to contact quorum of nodes, stepping down
2021/04/14 11:34:46 [INFO] raft: Node at 10.242.228.137:8300 [Follower] entering Follower state (Leader: "")
2021/04/14 11:34:46 [WARN] consul.coordinate: Batch update failed: leadership lost while committing log
2021/04/14 11:34:46 [INFO] raft: aborting pipeline replication to peer {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}
2021/04/14 11:34:46 [ERR] consul: failed to reconcile member: {dev-consul-server-2 10.242.70.73 8301 map[raft_vsn:3 port:8300 segment: id:590cff07-d5aa-e855-db2c-f428e810df3b vsn:2 build:1.0.6:9a494b5f role:consul vsn_max:3 expect:3 dc:dc1 wan_join_port:8302 vsn_min:2] failed 1 5 2 2 5 4}: leadership lost while committing log
2021/04/14 11:34:46 [INFO] consul: cluster leadership lost
2021/04/14 11:34:46 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:34:54 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/14 11:34:54 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 51
2021/04/14 11:34:54 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:34:54 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:34:54 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:34:54 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:50156->10.242.5.10:8300: i/o timeout
2021/04/14 11:34:54 [INFO] raft: Election won. Tally: 2
2021/04/14 11:34:54 [INFO] raft: Node at 10.242.228.137:8300 [Leader] entering Leader state
2021/04/14 11:34:54 [INFO] raft: Added peer 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, starting replication
2021/04/14 11:34:54 [INFO] raft: Added peer 590cff07-d5aa-e855-db2c-f428e810df3b, starting replication
2021/04/14 11:34:54 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:34:54 [INFO] consul: cluster leadership acquired
2021/04/14 11:34:54 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:34:54 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/14 11:34:54 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:34:54 [WARN] raft: AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300} rejected, sending older logs (next: 672)
2021/04/14 11:34:55 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:34:55 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:34:55 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.70.73
2021/04/14 11:34:55 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:34:56 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:34:56 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/14 11:34:56 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:50198->10.242.5.10:8300: i/o timeout
2021/04/14 11:34:57 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 2.502626947s
2021/04/14 11:34:58 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/14 11:34:58 [INFO] raft: pipelining replication to peer {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}
2021/04/14 11:34:58 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/14 11:34:58 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 4.00331533s
2021/04/14 11:35:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:47754->10.242.70.73:8301: i/o timeout
2021/04/14 11:35:01 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:35:01 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 6.320181262s
2021/04/14 11:35:01 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:35:01 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:35:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:35:03 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:35:04 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:50384->10.242.5.10:8300: i/o timeout
2021/04/14 11:35:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:35:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/14 11:35:04 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:50396->10.242.5.10:8300: i/o timeout
2021/04/14 11:35:04 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:35:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:47880->10.242.70.73:8301: i/o timeout
2021/04/14 11:35:05 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:35:05 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:50402->10.242.5.10:8300: i/o timeout
2021/04/14 11:35:05 [INFO] raft: aborting pipeline replication to peer {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}
2021/04/14 11:35:05 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: EOF
2021/04/14 11:35:05 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:35:05 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: read tcp 10.242.228.137:42420->10.242.70.73:8300: read: connection reset by peer
2021/04/14 11:35:05 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/14 11:35:05 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:35:06 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:35:06 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:35:06 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:35:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:35:06 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:35:06 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:35:06 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:35:06 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:35:07 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:35:07 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:35:07 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:35:07 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:35:07 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:35:07 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 2.504480976s
2021/04/14 11:35:07 [WARN] raft: Failed to contact quorum of nodes, stepping down
2021/04/14 11:35:07 [INFO] raft: Node at 10.242.228.137:8300 [Follower] entering Follower state (Leader: "")
2021/04/14 11:35:07 [ERR] memberlist: failed to receive: read tcp 10.242.228.137:8301->10.242.5.10:33060: i/o timeout from=10.242.5.10:33060
2021/04/14 11:35:07 [ERR] memberlist: Failed to send error: write tcp 10.242.228.137:8301->10.242.5.10:33060: i/o timeout from=10.242.5.10:33060
2021/04/14 11:35:07 [WARN] consul.coordinate: Batch update failed: leadership lost while committing log
2021/04/14 11:35:07 [INFO] consul: cluster leadership lost
2021/04/14 11:35:08 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:35:09 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:35:09 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:35:09 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.70.73
2021/04/14 11:35:09 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:35:09 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:35:09 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:35:11 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:35:11 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.70.73:8301
2021/04/14 11:35:14 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/14 11:35:14 [ERR] raft: Failed to AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:50602->10.242.5.10:8300: i/o timeout
2021/04/14 11:35:16 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:38902
2021/04/14 11:35:16 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:50666->10.242.5.10:8300: i/o timeout
2021/04/14 11:35:16 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/14 11:35:16 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 52
2021/04/14 11:35:16 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:35:16 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:35:16 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:35:18 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:38946
2021/04/14 11:35:19 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:42008
2021/04/14 11:35:23 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:39032
2021/04/14 11:35:25 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:35:25 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 53
2021/04/14 11:35:25 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:35:25 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:35:25 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:35:26 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:50926->10.242.5.10:8300: i/o timeout
2021/04/14 11:35:28 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:39096
2021/04/14 11:35:28 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/14 11:35:30 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:35:32 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:35:32 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 54
2021/04/14 11:35:32 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:35:32 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:35:32 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:35:35 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:51160->10.242.5.10:8300: i/o timeout
2021/04/14 11:35:36 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:39210
2021/04/14 11:35:38 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:35:38 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 55
2021/04/14 11:35:38 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:35:38 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:35:38 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:35:38 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:39252
2021/04/14 11:35:40 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:35:40 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/14 11:35:41 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:35:42 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:51286->10.242.5.10:8300: i/o timeout
2021/04/14 11:35:44 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:35:44 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.70.73
2021/04/14 11:35:44 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/14 11:35:46 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:39364
2021/04/14 11:35:46 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:35:46 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 56
2021/04/14 11:35:46 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:35:46 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:35:46 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:35:48 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:51446->10.242.5.10:8300: i/o timeout
2021/04/14 11:35:48 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:39398
2021/04/14 11:35:48 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/14 11:35:53 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:39494
2021/04/14 11:35:55 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:35:55 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 57
2021/04/14 11:35:55 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:35:55 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:35:55 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:35:56 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:51652->10.242.5.10:8300: i/o timeout
2021/04/14 11:35:57 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:35:58 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:39554
2021/04/14 11:36:03 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:36:03 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 58
2021/04/14 11:36:03 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:36:03 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:36:03 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:36:05 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:51884->10.242.5.10:8300: i/o timeout
2021/04/14 11:36:06 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:39658
2021/04/14 11:36:08 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:39694
2021/04/14 11:36:08 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.5.10:8302
2021/04/14 11:36:12 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:36:12 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 59
2021/04/14 11:36:12 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:36:12 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:36:12 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:36:13 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:52038->10.242.5.10:8300: i/o timeout
2021/04/14 11:36:16 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:39810
2021/04/14 11:36:18 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:39846
2021/04/14 11:36:21 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:36:22 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:36:22 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:36:22 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 60
2021/04/14 11:36:22 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:36:22 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:36:22 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:36:22 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:52246->10.242.5.10:8300: i/o timeout
2021/04/14 11:36:26 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:39956
2021/04/14 11:36:28 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:39992
2021/04/14 11:36:32 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:36:32 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 61
2021/04/14 11:36:32 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:36:32 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:36:32 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:36:32 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:52474->10.242.5.10:8300: i/o timeout
2021/04/14 11:36:35 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:40090
2021/04/14 11:36:38 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:40142
2021/04/14 11:36:41 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:36:41 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 62
2021/04/14 11:36:41 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:36:41 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:36:41 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:36:42 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:52652->10.242.5.10:8300: i/o timeout
2021/04/14 11:36:43 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:40220
2021/04/14 11:36:48 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:36:48 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 63
2021/04/14 11:36:48 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:36:48 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:36:48 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:36:48 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:40290
2021/04/14 11:36:49 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:44084
2021/04/14 11:36:51 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:36:51 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:52882->10.242.5.10:8300: i/o timeout
2021/04/14 11:36:54 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:40390
2021/04/14 11:36:56 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:36:56 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 64
2021/04/14 11:36:56 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:36:56 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:36:56 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:36:58 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:53032->10.242.5.10:8300: i/o timeout
2021/04/14 11:36:58 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:40452
2021/04/14 11:36:59 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:37:01 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:37:04 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:37:04 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 65
2021/04/14 11:37:04 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:37:04 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:37:04 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:37:06 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:40558
2021/04/14 11:37:06 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:53242->10.242.5.10:8300: i/o timeout
2021/04/14 11:37:08 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:40602
2021/04/14 11:37:08 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.5.10:8302
2021/04/14 11:37:11 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:37:11 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 66
2021/04/14 11:37:11 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:37:11 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:37:11 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:37:13 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:40698
2021/04/14 11:37:14 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:37:14 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:53404->10.242.5.10:8300: i/o timeout
2021/04/14 11:37:17 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:37:17 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 67
2021/04/14 11:37:17 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:37:17 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:37:17 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:37:18 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:40770
2021/04/14 11:37:19 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:44780
2021/04/14 11:37:21 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:53570->10.242.5.10:8300: i/o timeout
2021/04/14 11:37:25 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:37:25 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 68
2021/04/14 11:37:25 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:37:25 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:37:25 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:37:25 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:37:26 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:40880
2021/04/14 11:37:27 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:53702->10.242.5.10:8300: i/o timeout
2021/04/14 11:37:28 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:40920
2021/04/14 11:37:32 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:37:32 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 69
2021/04/14 11:37:32 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:37:32 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:37:32 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:37:33 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:41010
2021/04/14 11:37:35 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:53898->10.242.5.10:8300: i/o timeout
2021/04/14 11:37:38 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:37:38 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:41076
2021/04/14 11:37:40 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:37:40 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 70
2021/04/14 11:37:40 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:37:40 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:37:40 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:37:41 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:37:42 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:54042->10.242.5.10:8300: i/o timeout
2021/04/14 11:37:46 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:41192
2021/04/14 11:37:48 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:41230
2021/04/14 11:37:48 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:37:48 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 71
2021/04/14 11:37:48 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:37:48 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:37:48 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:37:49 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:45450
2021/04/14 11:37:50 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:54208->10.242.5.10:8300: i/o timeout
2021/04/14 11:37:53 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:41328
2021/04/14 11:37:56 [WARN] memberlist: Refuting a dead message (from: dev-consul-server-1)
2021/04/14 11:37:56 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 11:37:56 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 11:37:57 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:37:57 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 72
2021/04/14 11:37:57 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:37:57 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:37:57 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:37:58 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:41398
2021/04/14 11:37:58 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:54390->10.242.5.10:8300: i/o timeout
2021/04/14 11:37:58 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:38:03 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:38:05 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:38:05 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 73
2021/04/14 11:38:05 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:05 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:38:05 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:06 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:41508
2021/04/14 11:38:07 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:54610->10.242.5.10:8300: i/o timeout
2021/04/14 11:38:08 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:41548
2021/04/14 11:38:10 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:38:10 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 74
2021/04/14 11:38:10 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:10 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:38:10 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:13 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:41642
2021/04/14 11:38:15 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:54764->10.242.5.10:8300: i/o timeout
2021/04/14 11:38:18 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:38:18 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 75
2021/04/14 11:38:18 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:18 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:38:18 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:18 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:41712
2021/04/14 11:38:18 [ERR] memberlist: failed to receive: read tcp 10.242.228.137:8301->10.242.5.10:34530: i/o timeout from=10.242.5.10:34530
2021/04/14 11:38:18 [ERR] memberlist: Failed to send error: write tcp 10.242.228.137:8301->10.242.5.10:34530: i/o timeout from=10.242.5.10:34530
2021/04/14 11:38:18 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.70.73
2021/04/14 11:38:18 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/14 11:38:19 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:38:19 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:38:19 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:46114
2021/04/14 11:38:19 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.70.73
2021/04/14 11:38:19 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:38:20 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:54898->10.242.5.10:8300: i/o timeout
2021/04/14 11:38:20 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:41750
2021/04/14 11:38:21 [INFO] raft: Election won. Tally: 2
2021/04/14 11:38:21 [INFO] raft: Node at 10.242.228.137:8300 [Leader] entering Leader state
2021/04/14 11:38:21 [INFO] raft: Added peer 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, starting replication
2021/04/14 11:38:21 [INFO] raft: Added peer 590cff07-d5aa-e855-db2c-f428e810df3b, starting replication
2021/04/14 11:38:21 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/14 11:38:21 [INFO] consul: cluster leadership acquired
2021/04/14 11:38:21 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:21 [WARN] raft: AppendEntries to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300} rejected, sending older logs (next: 660)
2021/04/14 11:38:21 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:21 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:21 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:21 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:38:22 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:22 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:22 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:22 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:23 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:23 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:23 [INFO] raft: pipelining replication to peer {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}
2021/04/14 11:38:24 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:24 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 2.504086728s
2021/04/14 11:38:24 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/14 11:38:24 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:24 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:24 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/14 11:38:24 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.70.73
2021/04/14 11:38:24 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.70.73:8300) (DC: dc1)
2021/04/14 11:38:24 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/14 11:38:25 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/14 11:38:25 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:25 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:26 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 4.440790055s
2021/04/14 11:38:26 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:26 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:26 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:27 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:27 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:27 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:27 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:27 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:38:27 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:38:27 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:27 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:28 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 6.226369123s
2021/04/14 11:38:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:37508->10.242.5.10:8301: i/o timeout
2021/04/14 11:38:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:38:28 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:37548->10.242.5.10:8301: i/o timeout
2021/04/14 11:38:29 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:38:29 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 7.383029515s
2021/04/14 11:38:29 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 2.500454066s
2021/04/14 11:38:29 [WARN] raft: Failed to contact quorum of nodes, stepping down
2021/04/14 11:38:29 [INFO] raft: Node at 10.242.228.137:8300 [Follower] entering Follower state (Leader: "")
2021/04/14 11:38:29 [INFO] raft: aborting pipeline replication to peer {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}
2021/04/14 11:38:29 [ERR] consul: failed to reconcile member: {dev-consul-server-2 10.242.70.73 8301 map[id:590cff07-d5aa-e855-db2c-f428e810df3b vsn:2 vsn_max:3 expect:3 role:consul dc:dc1 raft_vsn:3 segment: wan_join_port:8302 vsn_min:2 port:8300 build:1.0.6:9a494b5f] failed 1 5 2 2 5 4}: leadership lost while committing log
2021/04/14 11:38:29 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:29 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:29 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:38:29 [INFO] consul: cluster leadership lost
2021/04/14 11:38:29 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/14 11:38:31 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:31 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:37626->10.242.5.10:8301: i/o timeout
2021/04/14 11:38:32 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:38:32 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:32 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:37678->10.242.5.10:8301: i/o timeout
2021/04/14 11:38:34 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:38:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:37710->10.242.5.10:8301: i/o timeout
2021/04/14 11:38:36 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:38:36 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/14 11:38:36 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 76
2021/04/14 11:38:36 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:36 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:38 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:42060
2021/04/14 11:38:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:37810->10.242.5.10:8301: i/o timeout
2021/04/14 11:38:39 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:38:39 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:38:39 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:38:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:37828->10.242.5.10:8301: i/o timeout
2021/04/14 11:38:41 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:38:43 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/14 11:38:44 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:38:44 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 77
2021/04/14 11:38:44 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:44 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:38:44 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:46 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:42172
2021/04/14 11:38:46 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:55236->10.242.5.10:8300: i/o timeout
2021/04/14 11:38:48 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:42210
2021/04/14 11:38:48 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:38:48 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:38:49 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:46932
2021/04/14 11:38:49 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:38:49 [INFO] raft: Election won. Tally: 2
2021/04/14 11:38:49 [INFO] raft: Node at 10.242.228.137:8300 [Leader] entering Leader state
2021/04/14 11:38:49 [INFO] raft: Added peer 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, starting replication
2021/04/14 11:38:49 [INFO] raft: Added peer 590cff07-d5aa-e855-db2c-f428e810df3b, starting replication
2021/04/14 11:38:49 [INFO] consul: cluster leadership acquired
2021/04/14 11:38:49 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/14 11:38:49 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:49 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:49 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:49 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:49 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:49 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:49 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:49 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:49 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:49 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:50 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:50 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:50 [INFO] raft: pipelining replication to peer {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}
2021/04/14 11:38:50 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:50 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:50 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:50 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:50 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:50 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:50 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:50 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:51 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:51 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:51 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:51 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:52 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 2.502215973s
2021/04/14 11:38:52 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:52 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:52 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:52 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:52 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:53 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:53 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:53 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:38:53 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/14 11:38:53 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:38:54 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:54 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:54 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:54 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 4.992675909s
2021/04/14 11:38:55 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:55 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:55 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:55 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:55 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:38:56 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:56 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:56 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:57 [WARN] raft: Failed to contact 590cff07-d5aa-e855-db2c-f428e810df3b in 7.472388804s
2021/04/14 11:38:57 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:38:58 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:38:58 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:58 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:38:58 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/14 11:38:59 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:38:59 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/14 11:38:59 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:38:59 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.70.73
2021/04/14 11:38:59 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/14 11:39:00 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:39:00 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:00 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:39:00 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:00 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:38350->10.242.5.10:8301: i/o timeout
2021/04/14 11:39:01 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:39:01 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:02 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:03 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:39:03 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:03 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:04 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:05 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:06 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:07 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:08 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:09 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:39:09 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:09 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:10 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:10 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:39:10 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:11 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:11 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/14 11:39:11 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:39:12 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:13 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:13 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/14 11:39:14 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:15 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:39:15 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:39:16 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:17 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:17 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/14 11:39:18 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:19 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:39:19 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:38806->10.242.5.10:8301: i/o timeout
2021/04/14 11:39:20 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:39:20 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:39:20 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:39:20 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:20 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:39:20 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:21 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:38840->10.242.5.10:8301: i/o timeout
2021/04/14 11:39:22 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:39:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:39:22 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:23 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:39:23 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:39:23 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:39:23 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/14 11:39:23 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:39:24 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:25 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:25 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/14 11:39:26 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:39:26 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:39:26 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/14 11:39:26 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:27 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:28 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:29 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:29 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/14 11:39:30 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:31 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:39:31 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:31 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:39:31 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:31 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:32 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:33 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:34 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:35 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:35 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:39:36 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:37 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:38 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:38 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.70.73:8302
2021/04/14 11:39:39 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:40 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:41 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:39:41 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:41 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:41 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:39:42 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:39:42 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:42 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:43 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:44 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:45 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:46 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:47 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:47 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/14 11:39:48 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:49 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:50 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:51 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/14 11:39:51 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:51 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:39:51 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:39:52 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:39756->10.242.5.10:8301: i/o timeout
2021/04/14 11:39:53 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:39:53 [ERR] raft: Failed to heartbeat to 10.242.70.73:8300: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:53 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/14 11:39:54 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:39:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:39852->10.242.5.10:8301: i/o timeout
2021/04/14 11:39:55 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:39:55 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/14 11:39:56 [WARN] raft: Failed to contact 8ae7076f-56df-66c3-0a6a-390eb7f8d23e in 2.500319093s
2021/04/14 11:39:56 [WARN] raft: Failed to contact quorum of nodes, stepping down
2021/04/14 11:39:56 [INFO] raft: Node at 10.242.228.137:8300 [Follower] entering Follower state (Leader: "")
2021/04/14 11:39:56 [INFO] raft: aborting pipeline replication to peer {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}
2021/04/14 11:39:56 [WARN] consul.coordinate: Batch update failed: leadership lost while committing log
2021/04/14 11:39:56 [INFO] consul: cluster leadership lost
2021/04/14 11:39:56 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:39:56 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:39:56 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:39:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:39870->10.242.5.10:8301: i/o timeout
2021/04/14 11:39:57 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:39:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:41546->10.242.5.10:8302: i/o timeout
2021/04/14 11:39:58 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:40:01 [WARN] consul.coordinate: Batch update failed: node is not the leader
2021/04/14 11:40:01 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/14 11:40:01 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 78
2021/04/14 11:40:01 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:40:01 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:40:01 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:40:01 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:40:01 [ERR] raft: Failed to AppendEntries to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:40:04 [ERR] raft: Failed to heartbeat to 10.242.5.10:8300: read tcp 10.242.228.137:55948->10.242.5.10:8300: i/o timeout
2021/04/14 11:40:06 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:43456
2021/04/14 11:40:07 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:40:07 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 79
2021/04/14 11:40:07 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:40:07 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:40:07 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:40:08 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:43500
2021/04/14 11:40:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:41676->10.242.5.10:8302: i/o timeout
2021/04/14 11:40:10 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:40:11 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:57868->10.242.5.10:8300: i/o timeout
2021/04/14 11:40:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:41980->10.242.5.10:8302: i/o timeout
2021/04/14 11:40:13 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:40:14 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:43604
2021/04/14 11:40:15 [ERR] memberlist: failed to receive: read tcp 10.242.228.137:8301->10.242.5.10:36296: i/o timeout from=10.242.5.10:36296
2021/04/14 11:40:15 [ERR] memberlist: Failed to send error: write tcp 10.242.228.137:8301->10.242.5.10:36296: i/o timeout from=10.242.5.10:36296
2021/04/14 11:40:16 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:40:16 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 80
2021/04/14 11:40:16 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:40:16 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:40:16 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:40:17 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:58032->10.242.5.10:8300: i/o timeout
2021/04/14 11:40:18 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:43672
2021/04/14 11:40:19 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:49386
2021/04/14 11:40:21 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.70.73:8301
2021/04/14 11:40:23 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:40:23 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 81
2021/04/14 11:40:23 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:40:23 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:40:23 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:40:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:42116->10.242.5.10:8302: i/o timeout
2021/04/14 11:40:23 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:40:26 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:43790
2021/04/14 11:40:26 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:58318->10.242.5.10:8300: i/o timeout
2021/04/14 11:40:28 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:43830
2021/04/14 11:40:28 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:40:28 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 11:40:28 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 11:40:30 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:40:30 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 82
2021/04/14 11:40:30 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:40:30 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:40:30 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:40:33 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:58520->10.242.5.10:8300: i/o timeout
2021/04/14 11:40:33 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:40:36 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:43954
2021/04/14 11:40:37 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:40:37 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 83
2021/04/14 11:40:37 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:40:37 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:40:37 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:40:38 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:44000
2021/04/14 11:40:38 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.5.10:8302
2021/04/14 11:40:39 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:40:39 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.5.10:35312
2021/04/14 11:40:39 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:40:39 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/14 11:40:40 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:58754->10.242.5.10:8300: i/o timeout
2021/04/14 11:40:43 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:44092
2021/04/14 11:40:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:42572->10.242.5.10:8302: i/o timeout
2021/04/14 11:40:43 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:40:46 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:40:46 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 84
2021/04/14 11:40:46 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:40:46 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:40:46 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:40:47 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:58914->10.242.5.10:8300: i/o timeout
2021/04/14 11:40:48 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:44164
2021/04/14 11:40:49 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:40:49 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/14 11:40:49 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:40:49 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:50278
2021/04/14 11:40:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:41460->10.242.5.10:8301: i/o timeout
2021/04/14 11:40:52 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:40:52 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:40:52 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 85
2021/04/14 11:40:52 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:40:52 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:40:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:41588->10.242.5.10:8301: i/o timeout
2021/04/14 11:40:55 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:40:55 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:40:56 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:40:56 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:40:56 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:40:56 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:44298
2021/04/14 11:40:56 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:59196->10.242.5.10:8300: i/o timeout
2021/04/14 11:40:58 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:44338
2021/04/14 11:41:00 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:41:00 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 86
2021/04/14 11:41:00 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:41:00 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:41:00 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:41:02 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:59364->10.242.5.10:8300: i/o timeout
2021/04/14 11:41:04 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:44446
2021/04/14 11:41:05 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:41:05 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 87
2021/04/14 11:41:05 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:41:05 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:41:05 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:41:08 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:41:08 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:44504
2021/04/14 11:41:10 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:59576->10.242.5.10:8300: i/o timeout
2021/04/14 11:41:11 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:41:11 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 88
2021/04/14 11:41:12 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:41:12 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:41:12 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:41:15 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:59692->10.242.5.10:8300: i/o timeout
2021/04/14 11:41:16 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:44640
2021/04/14 11:41:18 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:44682
2021/04/14 11:41:19 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:51090
2021/04/14 11:41:20 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:41:20 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 89
2021/04/14 11:41:20 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:41:20 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:41:20 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:41:21 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:41:22 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:59902->10.242.5.10:8300: i/o timeout
2021/04/14 11:41:26 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:44802
2021/04/14 11:41:28 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:41:28 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 90
2021/04/14 11:41:28 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:41:28 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:41:28 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:41:28 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:44840
2021/04/14 11:41:29 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:41:30 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:60094->10.242.5.10:8300: i/o timeout
2021/04/14 11:41:34 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:41:34 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 91
2021/04/14 11:41:34 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:41:34 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:41:34 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:41:36 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:44972
2021/04/14 11:41:38 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:60342->10.242.5.10:8300: i/o timeout
2021/04/14 11:41:38 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:45020
2021/04/14 11:41:38 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.70.73:8302
2021/04/14 11:41:42 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:41:42 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:41:42 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:41:43 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:41:43 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 92
2021/04/14 11:41:43 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:41:43 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:41:43 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:45116
2021/04/14 11:41:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:42936->10.242.5.10:8301: i/o timeout
2021/04/14 11:41:44 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:41:44 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:60508->10.242.5.10:8300: i/o timeout
2021/04/14 11:41:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:42974->10.242.5.10:8301: i/o timeout
2021/04/14 11:41:46 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:41:48 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:41:48 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:41:48 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:41:48 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:45190
2021/04/14 11:41:49 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:51904
2021/04/14 11:41:52 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:41:52 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 93
2021/04/14 11:41:52 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:41:52 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:41:52 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:41:53 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:60772->10.242.5.10:8300: i/o timeout
2021/04/14 11:41:53 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:45306
2021/04/14 11:41:58 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:45366
2021/04/14 11:42:01 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.70.73:8301
2021/04/14 11:42:02 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:42:02 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:42:02 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:42:02 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:42:02 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 94
2021/04/14 11:42:02 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:42:02 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:60992->10.242.5.10:8300: i/o timeout
2021/04/14 11:42:06 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:45494
2021/04/14 11:42:08 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:45538
2021/04/14 11:42:08 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.5.10:8302
2021/04/14 11:42:08 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:42:08 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 95
2021/04/14 11:42:08 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:42:08 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:42:08 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:42:11 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:42:12 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:33032->10.242.5.10:8300: i/o timeout
2021/04/14 11:42:13 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:45646
2021/04/14 11:42:15 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:42:16 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:42:16 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 96
2021/04/14 11:42:16 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:42:16 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:42:16 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:42:18 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:45720
2021/04/14 11:42:18 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:33218->10.242.5.10:8300: i/o timeout
2021/04/14 11:42:19 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:52718
2021/04/14 11:42:20 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:45768
2021/04/14 11:42:22 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:42:22 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 97
2021/04/14 11:42:22 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:42:22 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:42:22 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:42:26 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:33410->10.242.5.10:8300: i/o timeout
2021/04/14 11:42:28 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:45886
2021/04/14 11:42:28 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:42:28 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 98
2021/04/14 11:42:28 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:42:28 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:42:28 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:42:31 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.70.73:8301
2021/04/14 11:42:32 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:33550->10.242.5.10:8300: i/o timeout
2021/04/14 11:42:32 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:45978
2021/04/14 11:42:33 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:42:35 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:42:35 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 99
2021/04/14 11:42:35 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:42:35 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:42:35 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:42:38 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:46092
2021/04/14 11:42:38 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:33764->10.242.5.10:8300: i/o timeout
2021/04/14 11:42:43 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:42:43 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 100
2021/04/14 11:42:43 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:42:43 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:42:43 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:42:44 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:42:45 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:33958->10.242.5.10:8300: i/o timeout
2021/04/14 11:42:46 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:46316
2021/04/14 11:42:48 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:46352
2021/04/14 11:42:48 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:42:48 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 101
2021/04/14 11:42:48 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:42:48 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:42:48 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:42:49 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:53580
2021/04/14 11:42:53 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:46456
2021/04/14 11:42:53 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:34206->10.242.5.10:8300: i/o timeout
2021/04/14 11:42:54 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:42:54 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 102
2021/04/14 11:42:54 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:42:54 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:42:54 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:42:56 [ERR] memberlist: failed to receive: read tcp 10.242.228.137:8302->10.242.5.10:54282: i/o timeout from=10.242.5.10:54282
2021/04/14 11:42:56 [ERR] memberlist: Failed to send error: write tcp 10.242.228.137:8302->10.242.5.10:54282: i/o timeout from=10.242.5.10:54282
2021/04/14 11:42:58 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:46538
2021/04/14 11:42:58 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:34348->10.242.5.10:8300: i/o timeout
2021/04/14 11:43:01 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:43:02 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:43:02 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:43:02 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 103
2021/04/14 11:43:02 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:43:02 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:43:02 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:43:04 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:34556->10.242.5.10:8300: i/o timeout
2021/04/14 11:43:06 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:46670
2021/04/14 11:43:07 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:43:07 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 104
2021/04/14 11:43:07 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:43:07 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:43:07 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:43:08 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:46714
2021/04/14 11:43:08 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.5.10:8302
2021/04/14 11:43:12 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:34744->10.242.5.10:8300: i/o timeout
2021/04/14 11:43:13 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:46830
2021/04/14 11:43:15 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:43:17 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:43:17 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 105
2021/04/14 11:43:17 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:43:17 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:43:17 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:43:17 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:34882->10.242.5.10:8300: i/o timeout
2021/04/14 11:43:18 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:46914
2021/04/14 11:43:19 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:54446
2021/04/14 11:43:26 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:47046
2021/04/14 11:43:26 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:43:26 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 106
2021/04/14 11:43:26 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:43:26 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:43:26 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:43:27 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:35160->10.242.5.10:8300: i/o timeout
2021/04/14 11:43:28 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:47088
2021/04/14 11:43:32 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:43:32 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 107
2021/04/14 11:43:32 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:43:32 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:43:32 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:43:33 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:47188
2021/04/14 11:43:34 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:43:36 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:35440->10.242.5.10:8300: i/o timeout
2021/04/14 11:43:38 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:43:38 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:43:38 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 108
2021/04/14 11:43:38 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:43:38 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:43:38 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:43:38 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:47268
2021/04/14 11:43:41 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:43:42 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:35574->10.242.5.10:8300: i/o timeout
2021/04/14 11:43:43 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:43:43 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 109
2021/04/14 11:43:43 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:43:43 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:43:43 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:43:46 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:47406
2021/04/14 11:43:48 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:35738->10.242.5.10:8300: i/o timeout
2021/04/14 11:43:48 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:47452
2021/04/14 11:43:49 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:43:49 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 110
2021/04/14 11:43:49 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:43:49 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:43:49 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:43:49 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:55284
2021/04/14 11:43:53 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:47556
2021/04/14 11:43:53 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:35920->10.242.5.10:8300: i/o timeout
2021/04/14 11:43:56 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:43:56 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 111
2021/04/14 11:43:56 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:43:56 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:43:56 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:43:58 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:47632
2021/04/14 11:43:59 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:36056->10.242.5.10:8300: i/o timeout
2021/04/14 11:44:01 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:44:05 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:44:05 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 112
2021/04/14 11:44:05 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:44:05 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:44:05 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:44:06 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:36286->10.242.5.10:8300: i/o timeout
2021/04/14 11:44:06 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:47770
2021/04/14 11:44:08 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:47812
2021/04/14 11:44:08 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.5.10:8302
2021/04/14 11:44:09 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:44:13 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:47928
2021/04/14 11:44:14 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:44:14 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 113
2021/04/14 11:44:14 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:44:14 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:44:14 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:44:15 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:36520->10.242.5.10:8300: i/o timeout
2021/04/14 11:44:18 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:48006
2021/04/14 11:44:19 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:44:19 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 114
2021/04/14 11:44:19 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:44:19 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:44:19 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:44:19 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:56150
2021/04/14 11:44:21 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.70.73:8301
2021/04/14 11:44:24 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:36790->10.242.5.10:8300: i/o timeout
2021/04/14 11:44:24 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:44:24 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 115
2021/04/14 11:44:24 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:44:24 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:44:24 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:44:25 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:44:26 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:48140
2021/04/14 11:44:28 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:48178
2021/04/14 11:44:29 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:36920->10.242.5.10:8300: i/o timeout
2021/04/14 11:44:30 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:44:30 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 116
2021/04/14 11:44:30 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:44:30 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:44:30 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:44:33 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:48290
2021/04/14 11:44:34 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:37118->10.242.5.10:8300: i/o timeout
2021/04/14 11:44:38 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:48362
2021/04/14 11:44:38 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:44:38 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 117
2021/04/14 11:44:38 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:44:38 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:44:38 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:44:40 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:37258->10.242.5.10:8300: i/o timeout
2021/04/14 11:44:41 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:44:46 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:44:46 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:48502
2021/04/14 11:44:47 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:44:47 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 118
2021/04/14 11:44:47 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:44:47 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:44:47 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:44:48 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:48540
2021/04/14 11:44:48 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:37480->10.242.5.10:8300: i/o timeout
2021/04/14 11:44:49 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:57018
2021/04/14 11:44:51 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:44:53 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:44:53 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 119
2021/04/14 11:44:53 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:44:53 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:44:53 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:44:53 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:48656
2021/04/14 11:44:57 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:37732->10.242.5.10:8300: i/o timeout
2021/04/14 11:44:58 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:48724
2021/04/14 11:45:00 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:48778
2021/04/14 11:45:01 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:45:01 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 120
2021/04/14 11:45:01 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:45:01 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:45:01 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:45:02 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:45:03 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:37896->10.242.5.10:8300: i/o timeout
2021/04/14 11:45:07 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:45:07 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 121
2021/04/14 11:45:07 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:45:07 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:45:07 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:45:08 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:48906
2021/04/14 11:45:08 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 11:45:08 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 11:45:11 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:38130->10.242.5.10:8300: i/o timeout
2021/04/14 11:45:11 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:45:14 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/04/14 11:45:15 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:45:15 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 122
2021/04/14 11:45:15 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:45:15 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:45:15 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:45:16 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:49052
2021/04/14 11:45:17 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:38288->10.242.5.10:8300: i/o timeout
2021/04/14 11:45:18 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:49094
2021/04/14 11:45:19 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:57884
2021/04/14 11:45:23 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:45:23 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 123
2021/04/14 11:45:23 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:45:23 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:45:23 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:45:25 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:38552->10.242.5.10:8300: i/o timeout
2021/04/14 11:45:26 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:49228
2021/04/14 11:45:28 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:45:28 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:49266
2021/04/14 11:45:31 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.70.73:8301
2021/04/14 11:45:33 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:45:33 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 124
2021/04/14 11:45:33 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:45:33 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:45:33 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:45:33 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:38786->10.242.5.10:8300: i/o timeout
2021/04/14 11:45:36 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:49408
2021/04/14 11:45:38 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:49454
2021/04/14 11:45:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:50738->10.242.5.10:8302: i/o timeout
2021/04/14 11:45:38 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:45:41 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:45:41 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 125
2021/04/14 11:45:41 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:45:41 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:45:41 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:45:43 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:39072->10.242.5.10:8300: i/o timeout
2021/04/14 11:45:46 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:49598
2021/04/14 11:45:46 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:45:47 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:45:47 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 126
2021/04/14 11:45:48 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:45:48 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:45:48 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:45:48 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:49634
2021/04/14 11:45:49 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:58744
2021/04/14 11:45:50 [ERR] memberlist: Push/Pull with dev-consul-server-1.dc1 failed: read tcp 10.242.228.137:50666->10.242.5.10:8302: i/o timeout
2021/04/14 11:45:51 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:39322->10.242.5.10:8300: i/o timeout
2021/04/14 11:45:54 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:45:54 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 127
2021/04/14 11:45:54 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:45:54 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:45:54 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:45:56 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:49788
2021/04/14 11:45:58 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:39522->10.242.5.10:8300: i/o timeout
2021/04/14 11:45:58 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:49840
2021/04/14 11:46:01 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:46:01 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 128
2021/04/14 11:46:01 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:46:01 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:46:01 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:46:01 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.70.73:8301
2021/04/14 11:46:03 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:49986
2021/04/14 11:46:03 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:46:04 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:39776->10.242.5.10:8300: i/o timeout
2021/04/14 11:46:08 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:50100
2021/04/14 11:46:08 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.70.73:8302
2021/04/14 11:46:10 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:46:10 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 129
2021/04/14 11:46:10 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:46:10 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:46:10 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:46:10 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:46:11 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:40002->10.242.5.10:8300: i/o timeout
2021/04/14 11:46:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:51856->10.242.5.10:8302: i/o timeout
2021/04/14 11:46:14 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:46:16 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:50318
2021/04/14 11:46:17 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:46:17 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 130
2021/04/14 11:46:17 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:46:17 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:46:17 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:46:18 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:50372
2021/04/14 11:46:19 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:59788
2021/04/14 11:46:20 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:40302->10.242.5.10:8300: i/o timeout
2021/04/14 11:46:26 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:50558
2021/04/14 11:46:27 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:46:27 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 131
2021/04/14 11:46:27 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:46:27 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:46:27 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:46:27 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:40520->10.242.5.10:8300: i/o timeout
2021/04/14 11:46:28 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:50596
2021/04/14 11:46:31 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:46:35 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:46:35 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 132
2021/04/14 11:46:35 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:46:35 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:46:35 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:46:36 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:50796
2021/04/14 11:46:36 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:46:37 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:40870->10.242.5.10:8300: i/o timeout
2021/04/14 11:46:38 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:50842
2021/04/14 11:46:38 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.70.73:8302
2021/04/14 11:46:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:52514->10.242.5.10:8302: i/o timeout
2021/04/14 11:46:39 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:46:39 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:46:41 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:46:41 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 133
2021/04/14 11:46:41 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:46:41 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:46:41 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:46:43 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:50960
2021/04/14 11:46:44 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:46:44 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 11:46:44 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 11:46:45 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:41116->10.242.5.10:8300: i/o timeout
2021/04/14 11:46:48 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:51050
2021/04/14 11:46:49 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:46:49 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 134
2021/04/14 11:46:49 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:46:49 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:46:49 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:46:49 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:60768
2021/04/14 11:46:51 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:41320->10.242.5.10:8300: i/o timeout
2021/04/14 11:46:53 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:51226
2021/04/14 11:46:56 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:46:56 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.5.10:39166
2021/04/14 11:46:56 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:46:56 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 135
2021/04/14 11:46:56 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:46:56 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:46:56 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:46:58 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:51304
2021/04/14 11:46:59 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:41528->10.242.5.10:8300: i/o timeout
2021/04/14 11:47:06 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:51462
2021/04/14 11:47:06 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:47:06 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 136
2021/04/14 11:47:06 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:47:06 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:47:06 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:47:06 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:41768->10.242.5.10:8300: i/o timeout
2021/04/14 11:47:08 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:51516
2021/04/14 11:47:08 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.70.73:8302
2021/04/14 11:47:11 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.70.73:8301
2021/04/14 11:47:13 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:47:13 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:51642
2021/04/14 11:47:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:53484->10.242.5.10:8302: i/o timeout
2021/04/14 11:47:13 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:47:14 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:47:14 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:47:14 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 137
2021/04/14 11:47:14 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:47:14 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:47:14 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:47:16 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:42034->10.242.5.10:8300: i/o timeout
2021/04/14 11:47:18 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:51736
2021/04/14 11:47:19 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:33416
2021/04/14 11:47:22 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:47:22 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 138
2021/04/14 11:47:22 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:47:22 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:47:22 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:47:24 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:42310->10.242.5.10:8300: i/o timeout
2021/04/14 11:47:26 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:51898
2021/04/14 11:47:28 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:51938
2021/04/14 11:47:28 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:47:28 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 139
2021/04/14 11:47:28 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:47:28 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:47:28 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:47:32 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:42524->10.242.5.10:8300: i/o timeout
2021/04/14 11:47:33 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:52056
2021/04/14 11:47:37 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:47:37 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 140
2021/04/14 11:47:37 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:47:37 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:47:37 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:47:38 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:47:38 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:52194
2021/04/14 11:47:38 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:42712->10.242.5.10:8300: i/o timeout
2021/04/14 11:47:38 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.70.73:8302
2021/04/14 11:47:41 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.70.73:8301
2021/04/14 11:47:43 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:47:43 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 141
2021/04/14 11:47:43 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:47:43 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:47:43 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:47:46 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:52336
2021/04/14 11:47:47 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:42986->10.242.5.10:8300: i/o timeout
2021/04/14 11:47:47 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:47:48 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:52376
2021/04/14 11:47:49 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:47:49 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 142
2021/04/14 11:47:49 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:47:49 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:47:49 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:47:49 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:34338
2021/04/14 11:47:50 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:47:53 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:52502
2021/04/14 11:47:53 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:43202->10.242.5.10:8300: i/o timeout
2021/04/14 11:47:57 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:47:57 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 143
2021/04/14 11:47:57 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:47:57 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:47:57 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:47:58 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:52582
2021/04/14 11:47:59 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:43330->10.242.5.10:8300: i/o timeout
2021/04/14 11:48:00 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:48:02 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:48:02 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 144
2021/04/14 11:48:02 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:48:02 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:48:02 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:48:06 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:52738
2021/04/14 11:48:07 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:43604->10.242.5.10:8300: i/o timeout
2021/04/14 11:48:08 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:52790
2021/04/14 11:48:08 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.5.10:8302
2021/04/14 11:48:08 [ERR] memberlist: failed to receive: read tcp 10.242.228.137:8302->10.242.5.10:56482: i/o timeout from=10.242.5.10:56482
2021/04/14 11:48:08 [ERR] memberlist: Failed to send error: write tcp 10.242.228.137:8302->10.242.5.10:56482: i/o timeout from=10.242.5.10:56482
2021/04/14 11:48:10 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:48:10 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 145
2021/04/14 11:48:10 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:48:10 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:48:10 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:48:11 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.70.73:8301
2021/04/14 11:48:12 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:43736->10.242.5.10:8300: i/o timeout
2021/04/14 11:48:12 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 11:48:12 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 11:48:13 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:52916
2021/04/14 11:48:17 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:48:17 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 146
2021/04/14 11:48:17 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:48:17 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:48:17 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:48:18 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:53008
2021/04/14 11:48:19 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:35248
2021/04/14 11:48:20 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:44006->10.242.5.10:8300: i/o timeout
2021/04/14 11:48:20 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:53062
2021/04/14 11:48:22 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:48:23 [ERR] memberlist: failed to receive: read tcp 10.242.228.137:8301->10.242.5.10:40208: i/o timeout from=10.242.5.10:40208
2021/04/14 11:48:23 [ERR] memberlist: Failed to send error: write tcp 10.242.228.137:8301->10.242.5.10:40208: i/o timeout from=10.242.5.10:40208
2021/04/14 11:48:26 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:48:26 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 147
2021/04/14 11:48:26 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:48:26 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:48:26 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:48:27 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:44212->10.242.5.10:8300: i/o timeout
2021/04/14 11:48:28 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:53234
2021/04/14 11:48:32 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:48:32 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:48:32 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 148
2021/04/14 11:48:32 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:48:32 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:48:32 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:48:35 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:53384
2021/04/14 11:48:36 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:44506->10.242.5.10:8300: i/o timeout
2021/04/14 11:48:38 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:53448
2021/04/14 11:48:38 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:48:38 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 149
2021/04/14 11:48:38 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:48:38 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:48:38 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:48:41 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:48:42 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:44738->10.242.5.10:8300: i/o timeout
2021/04/14 11:48:43 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.70.73:8302
2021/04/14 11:48:46 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:53612
2021/04/14 11:48:48 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:48:48 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 150
2021/04/14 11:48:48 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:48:48 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:48:48 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:48:48 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:53662
2021/04/14 11:48:48 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:44892->10.242.5.10:8300: i/o timeout
2021/04/14 11:48:48 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/04/14 11:48:49 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:36216
2021/04/14 11:48:51 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:48:53 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:48:53 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 151
2021/04/14 11:48:53 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:48:53 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:48:53 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:48:53 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:53784
2021/04/14 11:48:58 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:45182->10.242.5.10:8300: i/o timeout
2021/04/14 11:48:58 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:53852
2021/04/14 11:48:59 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:48:59 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 152
2021/04/14 11:48:59 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:48:59 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:48:59 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:49:01 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/04/14 11:49:03 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:45350->10.242.5.10:8300: i/o timeout
2021/04/14 11:49:03 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:49:06 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:53992
2021/04/14 11:49:06 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:49:06 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 153
2021/04/14 11:49:06 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:49:06 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:49:06 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:49:08 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:54040
2021/04/14 11:49:09 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:45548->10.242.5.10:8300: i/o timeout
2021/04/14 11:49:13 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:54180
2021/04/14 11:49:13 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:49:13 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 154
2021/04/14 11:49:13 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:49:13 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:49:13 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:49:16 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:45748->10.242.5.10:8300: i/o timeout
2021/04/14 11:49:18 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:54274
2021/04/14 11:49:19 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:37126
2021/04/14 11:49:20 [ERR] memberlist: Push/Pull with dev-consul-server-1.dc1 failed: read tcp 10.242.228.137:57246->10.242.5.10:8302: i/o timeout
2021/04/14 11:49:21 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:49:22 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:49:22 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 155
2021/04/14 11:49:22 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:49:22 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:49:22 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:49:23 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:45988->10.242.5.10:8300: i/o timeout
2021/04/14 11:49:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:57488->10.242.5.10:8302: i/o timeout
2021/04/14 11:49:23 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:49:25 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:49:26 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:54430
2021/04/14 11:49:28 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:49:28 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 156
2021/04/14 11:49:28 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:49:28 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:49:28 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:49:28 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:54476
2021/04/14 11:49:31 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:49:32 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:46236->10.242.5.10:8300: i/o timeout
2021/04/14 11:49:33 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:54606
2021/04/14 11:49:35 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:49:35 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 157
2021/04/14 11:49:35 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:49:35 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:49:35 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:49:38 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:46424->10.242.5.10:8300: i/o timeout
2021/04/14 11:49:38 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:54690
2021/04/14 11:49:45 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:49:45 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 158
2021/04/14 11:49:45 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:49:45 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:49:45 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:49:45 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:46660->10.242.5.10:8300: i/o timeout
2021/04/14 11:49:46 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:54848
2021/04/14 11:49:48 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:54892
2021/04/14 11:49:49 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:38068
2021/04/14 11:49:51 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:49:51 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 159
2021/04/14 11:49:51 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:49:51 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:49:51 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:49:53 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:55018
2021/04/14 11:49:53 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:49:53 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 11:49:54 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 11:49:55 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:46986->10.242.5.10:8300: i/o timeout
2021/04/14 11:49:58 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:55112
2021/04/14 11:50:00 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:50:00 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:50:01 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:50:01 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 160
2021/04/14 11:50:01 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:50:01 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:50:01 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:50:01 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:47144->10.242.5.10:8300: i/o timeout
2021/04/14 11:50:01 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.70.73:8301
2021/04/14 11:50:06 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:55250
2021/04/14 11:50:08 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:55296
2021/04/14 11:50:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:58578->10.242.5.10:8302: i/o timeout
2021/04/14 11:50:08 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:50:09 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:50:09 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 161
2021/04/14 11:50:09 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:50:09 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:50:09 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:50:11 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:47448->10.242.5.10:8300: i/o timeout
2021/04/14 11:50:13 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.70.73:8302
2021/04/14 11:50:13 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:55418
2021/04/14 11:50:15 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:50:15 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 162
2021/04/14 11:50:15 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:50:15 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:50:15 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:50:18 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:55512
2021/04/14 11:50:19 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:47692->10.242.5.10:8300: i/o timeout
2021/04/14 11:50:19 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:38962
2021/04/14 11:50:21 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:50:21 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 163
2021/04/14 11:50:21 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:50:21 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:50:21 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: dial tcp ->10.242.70.73:8300: getsockopt: connection refused
2021/04/14 11:50:25 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:47862->10.242.5.10:8300: i/o timeout
2021/04/14 11:50:26 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:55668
2021/04/14 11:50:27 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:50:28 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:55712
2021/04/14 11:50:30 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:50:31 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:50:31 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 164
2021/04/14 11:50:31 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:50:31 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:50:31 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:48028->10.242.5.10:8300: i/o timeout
2021/04/14 11:50:31 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:50:32 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:50:32 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:50:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:58848->10.242.5.10:8301: i/o timeout
2021/04/14 11:50:34 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:50:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:58894->10.242.5.10:8301: i/o timeout
2021/04/14 11:50:36 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:50:36 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:55874
2021/04/14 11:50:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:58914->10.242.5.10:8301: i/o timeout
2021/04/14 11:50:38 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:50:38 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:50:38 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:50:38 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:50:38 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:55926
2021/04/14 11:50:40 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:50:40 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 165
2021/04/14 11:50:40 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:50:40 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:50:41 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:48374->10.242.5.10:8300: i/o timeout
2021/04/14 11:50:41 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:40180->10.242.70.73:8300: i/o timeout
2021/04/14 11:50:43 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.5.10:8302
2021/04/14 11:50:46 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:56088
2021/04/14 11:50:48 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:56130
2021/04/14 11:50:49 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:50:49 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 166
2021/04/14 11:50:49 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:50:49 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:50:49 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:39932
2021/04/14 11:50:50 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:40504->10.242.70.73:8300: i/o timeout
2021/04/14 11:50:50 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:48698->10.242.5.10:8300: i/o timeout
2021/04/14 11:50:52 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:50:56 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:56298
2021/04/14 11:50:57 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:50:57 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 167
2021/04/14 11:50:57 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:50:57 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:50:58 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:56344
2021/04/14 11:50:59 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:50:59 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:40742->10.242.70.73:8300: i/o timeout
2021/04/14 11:50:59 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:48936->10.242.5.10:8300: i/o timeout
2021/04/14 11:51:02 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.70.73:8301
2021/04/14 11:51:02 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:51:02 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 168
2021/04/14 11:51:03 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:51:03 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:51:06 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:56500
2021/04/14 11:51:07 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:49182->10.242.5.10:8300: i/o timeout
2021/04/14 11:51:07 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:40988->10.242.70.73:8300: i/o timeout
2021/04/14 11:51:08 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:56554
2021/04/14 11:51:11 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:51:11 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 169
2021/04/14 11:51:11 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:51:11 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:51:13 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:41122->10.242.70.73:8300: i/o timeout
2021/04/14 11:51:13 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:49316->10.242.5.10:8300: i/o timeout
2021/04/14 11:51:16 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:56712
2021/04/14 11:51:16 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:51:18 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:56760
2021/04/14 11:51:19 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:40784
2021/04/14 11:51:20 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:51:20 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 170
2021/04/14 11:51:20 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:51:20 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:51:21 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:41382->10.242.70.73:8300: i/o timeout
2021/04/14 11:51:21 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:49576->10.242.5.10:8300: i/o timeout
2021/04/14 11:51:26 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:56916
2021/04/14 11:51:27 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:51:28 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:56958
2021/04/14 11:51:28 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:51:30 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:51:30 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 171
2021/04/14 11:51:30 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:51:30 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:51:30 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:41626->10.242.70.73:8300: i/o timeout
2021/04/14 11:51:30 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:49820->10.242.5.10:8300: i/o timeout
2021/04/14 11:51:32 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.70.73:8301
2021/04/14 11:51:36 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:57118
2021/04/14 11:51:38 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:57170
2021/04/14 11:51:39 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:51:39 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 172
2021/04/14 11:51:39 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:51:39 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:51:40 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:50118->10.242.5.10:8300: i/o timeout
2021/04/14 11:51:40 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:41924->10.242.70.73:8300: i/o timeout
2021/04/14 11:51:43 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.5.10:8302
2021/04/14 11:51:43 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:57284
2021/04/14 11:51:44 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:51:44 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 173
2021/04/14 11:51:44 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:51:44 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:51:44 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:51:48 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:57378
2021/04/14 11:51:49 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:50388->10.242.5.10:8300: i/o timeout
2021/04/14 11:51:49 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:42194->10.242.70.73:8300: i/o timeout
2021/04/14 11:51:49 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:41654
2021/04/14 11:51:50 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:57438
2021/04/14 11:51:51 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:51:51 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 174
2021/04/14 11:51:51 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:51:51 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:51:54 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:50552->10.242.5.10:8300: i/o timeout
2021/04/14 11:51:54 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:42358->10.242.70.73:8300: i/o timeout
2021/04/14 11:51:57 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:57578
2021/04/14 11:51:58 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:57596
2021/04/14 11:51:58 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:52:00 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:52:00 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 175
2021/04/14 11:52:00 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:52:00 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:52:01 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:50702->10.242.5.10:8300: i/o timeout
2021/04/14 11:52:01 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:42508->10.242.70.73:8300: i/o timeout
2021/04/14 11:52:02 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:52:06 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:57748
2021/04/14 11:52:07 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/14 11:52:08 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:57798
2021/04/14 11:52:10 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:52:10 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 176
2021/04/14 11:52:10 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:52:10 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:52:10 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:50980->10.242.5.10:8300: i/o timeout
2021/04/14 11:52:10 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:42786->10.242.70.73:8300: i/o timeout
2021/04/14 11:52:13 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:57926
2021/04/14 11:52:15 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:52:15 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 177
2021/04/14 11:52:15 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:52:15 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:52:18 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:58018
2021/04/14 11:52:19 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:42504
2021/04/14 11:52:20 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:51258->10.242.5.10:8300: i/o timeout
2021/04/14 11:52:20 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:43064->10.242.70.73:8300: i/o timeout
2021/04/14 11:52:20 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:52:24 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:52:24 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 178
2021/04/14 11:52:24 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:52:24 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:52:25 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:51414->10.242.5.10:8300: i/o timeout
2021/04/14 11:52:25 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:43220->10.242.70.73:8300: i/o timeout
2021/04/14 11:52:26 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:58168
2021/04/14 11:52:28 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:58216
2021/04/14 11:52:30 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:52:30 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 179
2021/04/14 11:52:30 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:52:30 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:52:33 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:58336
2021/04/14 11:52:33 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:52:34 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:43510->10.242.70.73:8300: i/o timeout
2021/04/14 11:52:34 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:51700->10.242.5.10:8300: i/o timeout
2021/04/14 11:52:37 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:52:37 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 180
2021/04/14 11:52:37 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:52:37 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:52:38 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:58420
2021/04/14 11:52:40 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:43658->10.242.70.73:8300: i/o timeout
2021/04/14 11:52:40 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:51852->10.242.5.10:8300: i/o timeout
2021/04/14 11:52:40 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.5.10
2021/04/14 11:52:40 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:52:42 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.70.73:8301
2021/04/14 11:52:42 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 11:52:42 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 11:52:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:34454->10.242.5.10:8301: i/o timeout
2021/04/14 11:52:44 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:52:44 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:52:44 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 181
2021/04/14 11:52:44 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:52:46 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:58572
2021/04/14 11:52:46 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:52:46 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.5.10
2021/04/14 11:52:46 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.5.10:8300) (DC: dc1)
2021/04/14 11:52:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:34536->10.242.5.10:8301: i/o timeout
2021/04/14 11:52:47 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/14 11:52:47 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:52082->10.242.5.10:8300: i/o timeout
2021/04/14 11:52:47 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:43892->10.242.70.73:8300: i/o timeout
2021/04/14 11:52:48 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:58618
2021/04/14 11:52:49 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:43438
2021/04/14 11:52:50 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:52:50 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/14 11:52:53 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:52:53 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 182
2021/04/14 11:52:53 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:52:53 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:52:53 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:58754
2021/04/14 11:52:54 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:44140->10.242.70.73:8300: i/o timeout
2021/04/14 11:52:54 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:52330->10.242.5.10:8300: i/o timeout
2021/04/14 11:52:58 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:58834
2021/04/14 11:53:01 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:53:01 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 183
2021/04/14 11:53:01 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:53:01 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:53:03 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:52584->10.242.5.10:8300: i/o timeout
2021/04/14 11:53:03 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:44390->10.242.70.73:8300: i/o timeout
2021/04/14 11:53:04 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:53:06 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:58992
2021/04/14 11:53:08 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:59046
2021/04/14 11:53:10 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:53:10 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 184
2021/04/14 11:53:10 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:53:10 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:53:11 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:44634->10.242.70.73:8300: i/o timeout
2021/04/14 11:53:11 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:52828->10.242.5.10:8300: i/o timeout
2021/04/14 11:53:12 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:53:13 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:53:13 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:59176
2021/04/14 11:53:18 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:59292
2021/04/14 11:53:19 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:44370
2021/04/14 11:53:20 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:53:20 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 185
2021/04/14 11:53:20 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:53:20 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:53:20 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:53112->10.242.5.10:8300: i/o timeout
2021/04/14 11:53:20 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:44918->10.242.70.73:8300: i/o timeout
2021/04/14 11:53:25 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:53:25 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 186
2021/04/14 11:53:25 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:53:25 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:53:26 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:59446
2021/04/14 11:53:28 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:59492
2021/04/14 11:53:28 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/04/14 11:53:30 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:53384->10.242.5.10:8300: i/o timeout
2021/04/14 11:53:30 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:45194->10.242.70.73:8300: i/o timeout
2021/04/14 11:53:33 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:53:33 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:53:33 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 187
2021/04/14 11:53:33 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:53:33 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:53:33 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:59628
2021/04/14 11:53:35 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:45362->10.242.70.73:8300: i/o timeout
2021/04/14 11:53:35 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:53556->10.242.5.10:8300: i/o timeout
2021/04/14 11:53:38 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:59704
2021/04/14 11:53:40 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:53:40 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 188
2021/04/14 11:53:40 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:53:40 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:53:40 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:59770
2021/04/14 11:53:43 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:53794->10.242.5.10:8300: i/o timeout
2021/04/14 11:53:43 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:45604->10.242.70.73:8300: i/o timeout
2021/04/14 11:53:45 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:53:46 [ERR] yamux: keepalive failed: session shutdown
2021/04/14 11:53:48 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:59914
2021/04/14 11:53:48 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:53:48 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 189
2021/04/14 11:53:48 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:53:48 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:53:49 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:45240
2021/04/14 11:53:50 [ERR] memberlist: Push/Pull with dev-consul-server-1.dc1 failed: read tcp 10.242.228.137:37158->10.242.5.10:8302: i/o timeout
2021/04/14 11:53:50 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:54016->10.242.5.10:8300: i/o timeout
2021/04/14 11:53:50 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:45822->10.242.70.73:8300: i/o timeout
2021/04/14 11:53:52 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.5.10:8301
2021/04/14 11:53:55 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:60046
2021/04/14 11:53:55 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:53:55 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 190
2021/04/14 11:53:55 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:53:55 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:53:58 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:60112
2021/04/14 11:53:58 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:54218->10.242.5.10:8300: i/o timeout
2021/04/14 11:53:58 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:46024->10.242.70.73:8300: i/o timeout
2021/04/14 11:54:00 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:54:04 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:54:04 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 191
2021/04/14 11:54:04 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:54:04 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:54:05 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:46242->10.242.70.73:8300: i/o timeout
2021/04/14 11:54:05 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:54436->10.242.5.10:8300: i/o timeout
2021/04/14 11:54:06 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:60262
2021/04/14 11:54:08 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:60316
2021/04/14 11:54:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:37658->10.242.5.10:8302: i/o timeout
2021/04/14 11:54:08 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:54:10 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:54:11 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:54:11 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 192
2021/04/14 11:54:11 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:54:11 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:54:13 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.70.73:8302
2021/04/14 11:54:14 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:54676->10.242.5.10:8300: i/o timeout
2021/04/14 11:54:14 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:46482->10.242.70.73:8300: i/o timeout
2021/04/14 11:54:15 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:60462
2021/04/14 11:54:16 [ERR] memberlist: failed to receive: read tcp 10.242.228.137:8302->10.242.5.10:59118: i/o timeout from=10.242.5.10:59118
2021/04/14 11:54:16 [ERR] memberlist: Failed to send error: write tcp 10.242.228.137:8302->10.242.5.10:59118: i/o timeout from=10.242.5.10:59118
2021/04/14 11:54:17 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:54:17 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 193
2021/04/14 11:54:17 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:54:17 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:54:18 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:60532
2021/04/14 11:54:19 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:46068
2021/04/14 11:54:21 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:54864->10.242.5.10:8300: i/o timeout
2021/04/14 11:54:21 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:46670->10.242.70.73:8300: i/o timeout
2021/04/14 11:54:25 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:54:25 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 194
2021/04/14 11:54:25 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:54:25 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:54:26 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:60686
2021/04/14 11:54:27 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:46830->10.242.70.73:8300: i/o timeout
2021/04/14 11:54:27 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:55024->10.242.5.10:8300: i/o timeout
2021/04/14 11:54:28 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/14 11:54:28 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:60734
2021/04/14 11:54:32 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.70.73:8301
2021/04/14 11:54:33 [ERR] agent: Coordinate update error: No cluster leader
2021/04/14 11:54:33 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:54:33 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 195
2021/04/14 11:54:33 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:54:33 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:54:35 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:47086->10.242.70.73:8300: i/o timeout
2021/04/14 11:54:35 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:55280->10.242.5.10:8300: i/o timeout
2021/04/14 11:54:36 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:60888
2021/04/14 11:54:38 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:60942
2021/04/14 11:54:38 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/14 11:54:38 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.5.10
2021/04/14 11:54:38 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/14 11:54:40 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:54:40 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 196
2021/04/14 11:54:40 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:54:40 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:54:43 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.70.73:8302
2021/04/14 11:54:43 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.198.137:32830
2021/04/14 11:54:43 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:47328->10.242.70.73:8300: i/o timeout
2021/04/14 11:54:43 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:55522->10.242.5.10:8300: i/o timeout
2021/04/14 11:54:48 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:32916
2021/04/14 11:54:49 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.228.155:46930
2021/04/14 11:54:49 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:54:49 [INFO] raft: Node at 10.242.228.137:8300 [Candidate] entering Candidate state in term 197
2021/04/14 11:54:49 [WARN] Unable to get address for server id 590cff07-d5aa-e855-db2c-f428e810df3b, using fallback address 10.242.70.73:8300: Could not find address for server id 590cff07-d5aa-e855-db2c-f428e810df3b
2021/04/14 11:54:49 [WARN] Unable to get address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e, using fallback address 10.242.5.10:8300: Could not find address for server id 8ae7076f-56df-66c3-0a6a-390eb7f8d23e
2021/04/14 11:54:50 [ERR] raft: Failed to make RequestVote RPC to {Voter 590cff07-d5aa-e855-db2c-f428e810df3b 10.242.70.73:8300}: read tcp 10.242.228.137:47504->10.242.70.73:8300: i/o timeout
2021/04/14 11:54:50 [ERR] raft: Failed to make RequestVote RPC to {Voter 8ae7076f-56df-66c3-0a6a-390eb7f8d23e 10.242.5.10:8300}: read tcp 10.242.228.137:55694->10.242.5.10:8300: i/o timeout
2021/04/14 11:54:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.228.137:38792->10.242.5.10:8302: i/o timeout
2021/04/14 11:54:53 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/14 11:54:56 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.198.137:33076
2021/04/14 11:54:56 [WARN] raft: Election timeout reached, restarting election
2021/04/14 11:54:56 [INFO] raft: Node at 10.242.228.137:8300 [Candi