By type
+ CONSUL_BIND=
+ [ -n ]
+ CONSUL_CLIENT=
+ [ -n ]
+ CONSUL_DATA_DIR=/consul/data
+ CONSUL_CONFIG_DIR=/consul/config
+ [ -n ]
+ [ a = - ]
+ [ agent = agent ]
+ shift
+ set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
+ [ consul = consul ]
+ stat -c %u /consul/data
+ id -u consul
+ [ 100 != 100 ]
+ stat -c %u /consul/config
+ id -u consul
+ [ 100 != 100 ]
+ [ ! -z ]
+ exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
bootstrap_expect > 0: expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
Version: 'v1.0.6'
Node ID: '9959677e-074a-a336-5a52-0f08e5968308'
Node name: 'dev-consul-server-0'
Datacenter: 'dc1' (Segment: '')
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.242.1.135 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2021/05/21 10:47:21 [INFO] raft: Initial configuration (index=0): []
2021/05/21 10:47:21 [INFO] raft: Node at 10.242.1.135:8300 [Follower] entering Follower state (Leader: "")
2021/05/21 10:47:21 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.1.135
2021/05/21 10:47:21 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.1.135
2021/05/21 10:47:21 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/21 10:47:21 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2021/05/21 10:47:21 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2021/05/21 10:47:21 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.1.135:8300) (DC: dc1)
2021/05/21 10:47:21 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
2021/05/21 10:47:21 [INFO] agent: started state syncer
2021/05/21 10:47:21 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer
2021/05/21 10:47:21 [INFO] agent: Joining LAN cluster...
2021/05/21 10:47:21 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/05/21 10:47:21 [WARN] memberlist: Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
2021/05/21 10:47:21 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
2021/05/21 10:47:21 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/05/21 10:47:21 [INFO] agent: (LAN) joined: 0 Err: 3 error(s) occurred:
* Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/05/21 10:47:21 [WARN] agent: Join LAN failed: , retrying in 30s
2021/05/21 10:47:29 [ERR] agent: failed to sync remote state: No cluster leader
2021/05/21 10:47:30 [WARN] raft: no known peers, aborting election
==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=45370c1e-4bed-6367-9685-dba8dabf7d6b&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2021/05/21 10:47:51 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/05/21 10:47:51 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
2021/05/21 10:47:51 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/05/21 10:47:51 [INFO] agent: (LAN) joined: 1 Err:
2021/05/21 10:47:51 [INFO] agent: Join LAN completed. Synced with 1 initial agents
2021/05/21 10:47:55 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 10:47:55 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.129.75
2021/05/21 10:47:55 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/21 10:47:55 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.75
2021/05/21 10:47:55 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 10:47:56 [ERR] agent: Coordinate update error: No cluster leader
2021/05/21 10:48:01 [ERR] agent: failed to sync remote state: No cluster leader
2021/05/21 10:48:31 [ERR] agent: Coordinate update error: No cluster leader
2021/05/21 10:48:34 [ERR] agent: failed to sync remote state: No cluster leader
2021/05/21 10:48:57 [ERR] agent: failed to sync remote state: No cluster leader
2021/05/21 10:49:03 [ERR] agent: Coordinate update error: No cluster leader
2021/05/21 10:49:05 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.142.201
2021/05/21 10:49:05 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.142.201:8300) (DC: dc1)
2021/05/21 10:49:05 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.242.1.135:8300,10.242.129.75:8300,10.242.142.201:8300
2021/05/21 10:49:05 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.142.201
2021/05/21 10:49:05 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/05/21 10:49:07 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/05/21 10:49:07 [INFO] raft: Node at 10.242.1.135:8300 [Candidate] entering Candidate state in term 2
2021/05/21 10:49:07 [INFO] raft: Election won. Tally: 2
2021/05/21 10:49:07 [INFO] raft: Node at 10.242.1.135:8300 [Leader] entering Leader state
2021/05/21 10:49:07 [INFO] raft: Added peer 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, starting replication
2021/05/21 10:49:07 [INFO] raft: Added peer c955779f-89a6-203f-a38f-22883c86e868, starting replication
2021/05/21 10:49:07 [INFO] consul: cluster leadership acquired
2021/05/21 10:49:07 [INFO] consul: New leader elected: dev-consul-server-0
2021/05/21 10:49:07 [WARN] raft: AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300} rejected, sending older logs (next: 1)
2021/05/21 10:49:07 [INFO] raft: pipelining replication to peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}
2021/05/21 10:49:07 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' joined, marking health alive
2021/05/21 10:49:07 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/21 10:49:07 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/21 10:49:07 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/05/21 10:49:08 [INFO] raft: pipelining replication to peer {Voter c955779f-89a6-203f-a38f-22883c86e868 10.242.142.201:8300}
2021/05/21 10:49:10 [INFO] agent: Synced node info
2021/05/21 10:49:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:49:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:49:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:49:24 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:49:24 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 10:49:42 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:49:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:49:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:50:52 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:50:52 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 10:51:28 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:51:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:51:38 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:51:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:51:42 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:51:42 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 10:51:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:51:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:51:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:51:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:51:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:52:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:52:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:52:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:52:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:52:16 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:52:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:52:20 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:52:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:52:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:52:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:52:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:52:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:52:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:52:52 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:52:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:52:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:52:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:53:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:53:04 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:53:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:53:14 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:53:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:53:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:53:20 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:53:38 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:53:40 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:53:40 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 10:53:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:54:02 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:54:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:54:06 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:54:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:54:10 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:54:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:54:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:54:30 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:54:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:54:34 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:54:34 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 10:54:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:54:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:54:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:54:42 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:55:14 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:55:18 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:55:20 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:55:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:55:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:55:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:55:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:55:29 [INFO] raft: aborting pipeline replication to peer {Voter c955779f-89a6-203f-a38f-22883c86e868 10.242.142.201:8300}
2021/05/21 10:55:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:55:32 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:55:32 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 10:55:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:55:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:55:37 [INFO] raft: pipelining replication to peer {Voter c955779f-89a6-203f-a38f-22883c86e868 10.242.142.201:8300}
2021/05/21 10:55:38 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:55:38 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 10:55:38 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 10:55:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:55:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:55:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:55:48 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:55:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:55:54 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:55:56 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:55:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:56:00 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:56:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:56:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:56:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:56:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:56:28 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:56:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:56:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:56:44 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:56:44 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 10:56:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:57:18 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:57:48 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:57:48 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 10:57:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:58:04 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:58:04 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 10:58:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:58:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:58:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:58:14 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:58:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:58:38 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:58:44 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:58:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 10:59:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 10:59:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:00:24 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:00:34 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:00:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:00:48 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:00:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:00:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:00:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:00:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:00:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:01:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:01:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:01:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:01:07 [INFO] raft: aborting pipeline replication to peer {Voter c955779f-89a6-203f-a38f-22883c86e868 10.242.142.201:8300}
2021/05/21 11:01:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:01:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:01:11 [INFO] raft: pipelining replication to peer {Voter c955779f-89a6-203f-a38f-22883c86e868 10.242.142.201:8300}
2021/05/21 11:01:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:01:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:01:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:01:18 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:01:18 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:01:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:01:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:01:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:01:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:01:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:01:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:01:32 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:01:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:01:36 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:01:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:01:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:01:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:02:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:02:28 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:02:28 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:02:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:02:36 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:02:44 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:02:54 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:03:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:03:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:03:14 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:03:24 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:03:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:03:30 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:03:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:03:44 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:03:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:03:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:03:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:04:48 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:04:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:05:00 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:05:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:05:14 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:05:54 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:05:54 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:05:54 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:06:34 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:06:56 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:06:58 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:06:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:06:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:53444->10.242.129.75:8301: i/o timeout
2021/05/21 11:07:03 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:07:06 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.50021592s
2021/05/21 11:07:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:37656->10.242.129.72:8301: i/o timeout
2021/05/21 11:07:06 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:07:06 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:07:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:08 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.986027665s
2021/05/21 11:07:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:37686->10.242.129.72:8301: i/o timeout
2021/05/21 11:07:09 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:07:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:11 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 7.439317497s
2021/05/21 11:07:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56472->10.242.129.75:8302: i/o timeout
2021/05/21 11:07:11 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:07:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:53502->10.242.129.75:8301: i/o timeout
2021/05/21 11:07:11 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:07:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:12 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:42246->10.242.129.75:8300: i/o timeout
2021/05/21 11:07:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:53510->10.242.129.75:8301: i/o timeout
2021/05/21 11:07:13 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:07:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:14 [INFO] raft: aborting pipeline replication to peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}
2021/05/21 11:07:15 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/21 11:07:15 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.75
2021/05/21 11:07:15 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:07:15 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/21 11:07:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:37732->10.242.129.72:8301: i/o timeout
2021/05/21 11:07:15 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:07:15 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:07:15 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/21 11:07:16 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:07:16 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' failed, marking health critical
2021/05/21 11:07:16 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500237788s
2021/05/21 11:07:16 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:07:16 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:07:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:18 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.75
2021/05/21 11:07:18 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:07:18 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/21 11:07:19 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.955680855s
2021/05/21 11:07:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:53570->10.242.129.75:8301: i/o timeout
2021/05/21 11:07:20 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:07:21 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 7.41396664s
2021/05/21 11:07:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:53580->10.242.129.75:8301: i/o timeout
2021/05/21 11:07:22 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:07:23 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:47378->10.242.129.75:8300: i/o timeout
2021/05/21 11:07:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:24 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:47384->10.242.129.75:8300: i/o timeout
2021/05/21 11:07:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:53610->10.242.129.75:8301: i/o timeout
2021/05/21 11:07:25 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:07:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56590->10.242.129.75:8302: i/o timeout
2021/05/21 11:07:26 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:07:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:53624->10.242.129.75:8301: i/o timeout
2021/05/21 11:07:27 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:07:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:53642->10.242.129.75:8301: i/o timeout
2021/05/21 11:07:29 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:07:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:31 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/21 11:07:31 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.75
2021/05/21 11:07:31 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:07:31 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/21 11:07:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:34 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:47456->10.242.129.75:8300: i/o timeout
2021/05/21 11:07:34 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:07:34 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:47458->10.242.129.75:8300: i/o timeout
2021/05/21 11:07:34 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:07:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56666->10.242.129.75:8302: i/o timeout
2021/05/21 11:07:36 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:07:36 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:07:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:38 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.502613629s
2021/05/21 11:07:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:41 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.951981969s
2021/05/21 11:07:41 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.129.75
2021/05/21 11:07:41 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/21 11:07:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:43 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 7.377198381s
2021/05/21 11:07:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:44 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:47534->10.242.129.75:8300: i/o timeout
2021/05/21 11:07:44 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:07:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56704->10.242.129.75:8302: i/o timeout
2021/05/21 11:07:46 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:07:46 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:47546->10.242.129.75:8300: i/o timeout
2021/05/21 11:07:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:47 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:07:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:51 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.129.75
2021/05/21 11:07:51 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/21 11:07:51 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.75:8301
2021/05/21 11:07:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:54 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:07:54 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:47608->10.242.129.75:8300: i/o timeout
2021/05/21 11:07:54 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:07:54 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/21 11:07:54 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.75
2021/05/21 11:07:54 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:07:54 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/21 11:07:54 [DEBUG] raft-net: 10.242.1.135:8300 accepted connection from: 10.242.129.75:33482
2021/05/21 11:07:54 [WARN] raft: Rejecting vote request from 10.242.129.75:8300 since we have a leader: 10.242.1.135:8300
2021/05/21 11:07:55 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:07:55 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' joined, marking health alive
2021/05/21 11:07:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:53852->10.242.129.75:8301: i/o timeout
2021/05/21 11:07:56 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:07:57 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500350942s
2021/05/21 11:07:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:07:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:38070->10.242.129.72:8301: i/o timeout
2021/05/21 11:07:59 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:07:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:00 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.987335556s
2021/05/21 11:08:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:53888->10.242.129.75:8301: i/o timeout
2021/05/21 11:08:01 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:08:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:02 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 7.475959594s
2021/05/21 11:08:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:53908->10.242.129.75:8301: i/o timeout
2021/05/21 11:08:03 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:08:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:04 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:47636->10.242.129.75:8300: i/o timeout
2021/05/21 11:08:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:38160->10.242.129.72:8301: i/o timeout
2021/05/21 11:08:05 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:08:05 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:47692->10.242.129.75:8300: i/o timeout
2021/05/21 11:08:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56926->10.242.129.75:8302: i/o timeout
2021/05/21 11:08:06 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:08:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:38180->10.242.129.72:8301: i/o timeout
2021/05/21 11:08:07 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:08:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:08 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/21 11:08:08 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.75
2021/05/21 11:08:08 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:08:08 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/21 11:08:08 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/21 11:08:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:38216->10.242.129.72:8301: i/o timeout
2021/05/21 11:08:09 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:08:10 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:08:10 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:08:11 [INFO] memberlist: Marking dev-consul-7467bf7b46-22bkc as failed, suspect timeout reached (1 peer confirmations)
2021/05/21 11:08:11 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:08:11 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' failed, marking health critical
2021/05/21 11:08:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:14 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:47788->10.242.129.75:8300: i/o timeout
2021/05/21 11:08:14 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:08:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:16 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:47840->10.242.129.75:8300: i/o timeout
2021/05/21 11:08:16 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:08:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:24 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.75:8301
2021/05/21 11:08:24 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:47942->10.242.129.75:8300: i/o timeout
2021/05/21 11:08:24 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:08:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:57168->10.242.129.75:8302: i/o timeout
2021/05/21 11:08:26 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:08:26 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:47968->10.242.129.75:8300: i/o timeout
2021/05/21 11:08:27 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:08:27 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.129.75:45178
2021/05/21 11:08:27 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:08:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:29 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:08:29 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/21 11:08:32 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:08:32 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:08:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:34 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:48048->10.242.129.75:8300: i/o timeout
2021/05/21 11:08:35 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:08:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:57216->10.242.129.75:8302: i/o timeout
2021/05/21 11:08:36 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:08:37 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:48074->10.242.129.75:8300: i/o timeout
2021/05/21 11:08:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:38 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:08:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:40 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.75
2021/05/21 11:08:40 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:08:40 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/21 11:08:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:54370->10.242.129.75:8301: i/o timeout
2021/05/21 11:08:41 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:08:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:44 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/21 11:08:45 [WARN] raft: Rejecting vote request from 10.242.129.75:8300 since we have a leader: 10.242.1.135:8300
2021/05/21 11:08:45 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:08:45 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' joined, marking health alive
2021/05/21 11:08:45 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:48168->10.242.129.75:8300: i/o timeout
2021/05/21 11:08:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:54406->10.242.129.75:8301: i/o timeout
2021/05/21 11:08:46 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:08:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:38630->10.242.129.72:8301: i/o timeout
2021/05/21 11:08:49 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:08:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:08:50 [ERR] raft: peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300} has newer term, stopping replication
2021/05/21 11:08:50 [INFO] raft: Node at 10.242.1.135:8300 [Follower] entering Follower state (Leader: "")
2021/05/21 11:08:50 [INFO] raft: aborting pipeline replication to peer {Voter c955779f-89a6-203f-a38f-22883c86e868 10.242.142.201:8300}
2021/05/21 11:08:50 [WARN] consul.coordinate: Batch update failed: leadership lost while committing log
2021/05/21 11:08:50 [INFO] consul: cluster leadership lost
2021/05/21 11:08:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:38656->10.242.129.72:8301: i/o timeout
2021/05/21 11:08:51 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:08:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:54466->10.242.129.75:8301: i/o timeout
2021/05/21 11:08:54 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:08:56 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/05/21 11:08:56 [INFO] raft: Node at 10.242.1.135:8300 [Candidate] entering Candidate state in term 3
2021/05/21 11:08:56 [INFO] raft: Node at 10.242.1.135:8300 [Follower] entering Follower state (Leader: "")
2021/05/21 11:08:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:38700->10.242.129.72:8301: i/o timeout
2021/05/21 11:08:56 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:08:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:38724->10.242.129.72:8301: i/o timeout
2021/05/21 11:08:59 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:09:01 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:09:01 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/05/21 11:09:01 [INFO] raft: Node at 10.242.1.135:8300 [Candidate] entering Candidate state in term 5
2021/05/21 11:09:01 [INFO] raft: Election won. Tally: 2
2021/05/21 11:09:01 [INFO] raft: Node at 10.242.1.135:8300 [Leader] entering Leader state
2021/05/21 11:09:01 [INFO] raft: Added peer 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, starting replication
2021/05/21 11:09:01 [INFO] raft: Added peer c955779f-89a6-203f-a38f-22883c86e868, starting replication
2021/05/21 11:09:01 [INFO] consul: cluster leadership acquired
2021/05/21 11:09:01 [INFO] consul: New leader elected: dev-consul-server-0
2021/05/21 11:09:01 [INFO] raft: pipelining replication to peer {Voter c955779f-89a6-203f-a38f-22883c86e868 10.242.142.201:8300}
2021/05/21 11:09:01 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' failed, marking health critical
2021/05/21 11:09:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:54560->10.242.129.75:8301: i/o timeout
2021/05/21 11:09:03 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:09:04 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.501753602s
2021/05/21 11:09:04 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:09:04 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:09:05 [ERR] memberlist: failed to receive: read tcp 10.242.1.135:8301->10.242.129.72:55768: i/o timeout from=10.242.129.72:55768
2021/05/21 11:09:05 [ERR] memberlist: Failed to send error: write tcp 10.242.1.135:8301->10.242.129.72:55768: i/o timeout from=10.242.129.72:55768
2021/05/21 11:09:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:54564->10.242.129.75:8301: i/o timeout
2021/05/21 11:09:05 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:09:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:09:06 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.951962751s
2021/05/21 11:09:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:09:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:54606->10.242.129.75:8301: i/o timeout
2021/05/21 11:09:08 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:09:09 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 7.36189758s
2021/05/21 11:09:09 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:09:09 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' joined, marking health alive
2021/05/21 11:09:09 [WARN] raft: Rejecting vote request from 10.242.129.75:8300 since we have a leader: 10.242.1.135:8300
2021/05/21 11:09:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:09:10 [WARN] raft: AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300} rejected, sending older logs (next: 295)
2021/05/21 11:09:10 [INFO] raft: pipelining replication to peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}
2021/05/21 11:09:28 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:09:28 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:09:30 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:09:30 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:09:30 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:09:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:09:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:09:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:39134->10.242.129.72:8301: i/o timeout
2021/05/21 11:09:40 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:09:40 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:09:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:09:41 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500285255s
2021/05/21 11:09:46 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:09:46 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:09:46 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:09:48 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:09:48 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:09:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:09:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:10:00 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:10:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:55168->10.242.129.75:8301: i/o timeout
2021/05/21 11:10:01 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:10:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:10:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:10:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:10:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:10:04 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.50017854s
2021/05/21 11:10:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:55194->10.242.129.75:8301: i/o timeout
2021/05/21 11:10:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:10:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:55212->10.242.129.75:8301: i/o timeout
2021/05/21 11:10:06 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:10:06 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:10:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:39426->10.242.129.72:8301: i/o timeout
2021/05/21 11:10:07 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:10:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:10:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:10:07 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.50026541s
2021/05/21 11:10:09 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.094819646s
2021/05/21 11:10:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:39440->10.242.129.72:8301: i/o timeout
2021/05/21 11:10:09 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:10:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:10:10 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:10:11 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 6.347309013s
2021/05/21 11:10:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:55248->10.242.129.75:8301: i/o timeout
2021/05/21 11:10:11 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:10:11 [INFO] raft: aborting pipeline replication to peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}
2021/05/21 11:10:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:10:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:10:13 [INFO] raft: pipelining replication to peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}
2021/05/21 11:10:22 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:10:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:10:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:39576->10.242.129.72:8301: i/o timeout
2021/05/21 11:10:28 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:10:30 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:10:30 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:10:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:10:32 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:10:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:39618->10.242.129.72:8301: i/o timeout
2021/05/21 11:10:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:10:34 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:10:34 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:10:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:10:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:10:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:39662->10.242.129.72:8301: i/o timeout
2021/05/21 11:10:37 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:10:38 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:10:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:10:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:55468->10.242.129.75:8301: i/o timeout
2021/05/21 11:10:40 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:10:40 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:10:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:10:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:10:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:39702->10.242.129.72:8301: i/o timeout
2021/05/21 11:10:42 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:10:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:10:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:39714->10.242.129.72:8301: i/o timeout
2021/05/21 11:10:44 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:10:44 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:10:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:10:46 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/21 11:10:46 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:10:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:55530->10.242.129.75:8301: i/o timeout
2021/05/21 11:10:47 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:10:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:10:49 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500245943s
2021/05/21 11:10:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:39754->10.242.129.72:8301: i/o timeout
2021/05/21 11:10:49 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:10:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:10:51 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.915432732s
2021/05/21 11:10:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:58538->10.242.129.75:8302: i/o timeout
2021/05/21 11:10:51 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:10:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:10:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:55560->10.242.129.75:8301: i/o timeout
2021/05/21 11:10:52 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:10:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:10:53 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 7.107324297s
2021/05/21 11:10:54 [INFO] memberlist: Marking dev-consul-7467bf7b46-22bkc as failed, suspect timeout reached (1 peer confirmations)
2021/05/21 11:10:54 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:10:54 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' failed, marking health critical
2021/05/21 11:10:54 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:10:54 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.129.72:33742
2021/05/21 11:10:54 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:10:55 [ERR] memberlist: failed to receive: read tcp 10.242.1.135:8301->10.242.129.75:47054: i/o timeout from=10.242.129.75:47054
2021/05/21 11:10:55 [ERR] memberlist: Failed to send error: write tcp 10.242.1.135:8301->10.242.129.75:47054: i/o timeout from=10.242.129.75:47054
2021/05/21 11:10:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:39794->10.242.129.72:8301: i/o timeout
2021/05/21 11:10:55 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:10:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:10:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:10:58 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.50025592s
2021/05/21 11:10:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:55624->10.242.129.75:8301: i/o timeout
2021/05/21 11:10:58 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:10:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:55630->10.242.129.75:8301: i/o timeout
2021/05/21 11:11:00 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:11:00 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500327088s
2021/05/21 11:11:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:55660->10.242.129.75:8301: i/o timeout
2021/05/21 11:11:02 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:11:03 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.994975186s
2021/05/21 11:11:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:04 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/21 11:11:04 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.75
2021/05/21 11:11:04 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:11:04 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/21 11:11:04 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.75:8301
2021/05/21 11:11:05 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 7.472435462s
2021/05/21 11:11:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:06 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:11:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:08 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500301272s
2021/05/21 11:11:10 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:11:11 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.95195703s
2021/05/21 11:11:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:58700->10.242.129.75:8302: i/o timeout
2021/05/21 11:11:11 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:11:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:13 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 7.445487561s
2021/05/21 11:11:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:15 [INFO] raft: aborting pipeline replication to peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}
2021/05/21 11:11:15 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:11:16 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:48270->10.242.129.75:8300: i/o timeout
2021/05/21 11:11:17 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:11:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:18 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500225478s
2021/05/21 11:11:18 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/21 11:11:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:20 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-22bkc)
2021/05/21 11:11:20 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.75
2021/05/21 11:11:20 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:11:20 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/21 11:11:20 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.972305636s
2021/05/21 11:11:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:58734->10.242.129.75:8302: i/o timeout
2021/05/21 11:11:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:55808->10.242.129.75:8301: i/o timeout
2021/05/21 11:11:22 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:11:23 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 7.419411989s
2021/05/21 11:11:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:55824->10.242.129.75:8301: i/o timeout
2021/05/21 11:11:24 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:11:25 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:49626->10.242.129.75:8300: i/o timeout
2021/05/21 11:11:26 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/21 11:11:26 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:11:27 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:49638->10.242.129.75:8300: i/o timeout
2021/05/21 11:11:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:55854->10.242.129.75:8301: i/o timeout
2021/05/21 11:11:27 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:11:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:55870->10.242.129.75:8301: i/o timeout
2021/05/21 11:11:30 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:11:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:55900->10.242.129.75:8301: i/o timeout
2021/05/21 11:11:32 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:11:33 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.75
2021/05/21 11:11:33 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:11:33 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/21 11:11:33 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:11:34 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:11:34 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:11:35 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:11:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:35 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:49704->10.242.129.75:8300: i/o timeout
2021/05/21 11:11:35 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:11:36 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:11:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:38 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500240902s
2021/05/21 11:11:38 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.75
2021/05/21 11:11:38 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:11:38 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/21 11:11:38 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:11:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:55966->10.242.129.75:8301: i/o timeout
2021/05/21 11:11:39 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:11:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:11:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:40 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:11:40 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.455936121s
2021/05/21 11:11:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:40194->10.242.129.72:8301: i/o timeout
2021/05/21 11:11:41 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:11:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:11:43 [INFO] raft: pipelining replication to peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}
2021/05/21 11:11:43 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' joined, marking health alive
2021/05/21 11:11:43 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/21 11:11:43 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.75
2021/05/21 11:11:43 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/21 11:11:43 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:11:43 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/21 11:11:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:11:44 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:11:44 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.75:8301
2021/05/21 11:11:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:40216->10.242.129.72:8301: i/o timeout
2021/05/21 11:11:44 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:11:44 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:11:45 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:11:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:11:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:46 [INFO] raft: aborting pipeline replication to peer {Voter c955779f-89a6-203f-a38f-22883c86e868 10.242.142.201:8300}
2021/05/21 11:11:46 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:11:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:40242->10.242.129.72:8301: i/o timeout
2021/05/21 11:11:47 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:11:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:11:48 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500216984s
2021/05/21 11:11:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:40254->10.242.129.72:8301: i/o timeout
2021/05/21 11:11:49 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:11:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:50 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:11:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:11:50 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:11:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:59042->10.242.129.75:8302: i/o timeout
2021/05/21 11:11:51 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:11:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:11:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:52 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500312254s
2021/05/21 11:11:53 [INFO] memberlist: Marking dev-consul-7467bf7b46-22bkc as failed, suspect timeout reached (1 peer confirmations)
2021/05/21 11:11:53 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:11:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:40294->10.242.129.72:8301: i/o timeout
2021/05/21 11:11:53 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:11:53 [INFO] raft: aborting pipeline replication to peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}
2021/05/21 11:11:53 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:11:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:11:55 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.75
2021/05/21 11:11:55 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:11:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:11:55 [INFO] raft: pipelining replication to peer {Voter c955779f-89a6-203f-a38f-22883c86e868 10.242.142.201:8300}
2021/05/21 11:11:55 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' failed, marking health critical
2021/05/21 11:11:56 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/21 11:11:56 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500242012s
2021/05/21 11:11:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56118->10.242.129.75:8301: i/o timeout
2021/05/21 11:11:56 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:11:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:11:57 [INFO] raft: pipelining replication to peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}
2021/05/21 11:12:00 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:12:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:12:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56174->10.242.129.75:8301: i/o timeout
2021/05/21 11:12:03 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:12:03 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500253521s
2021/05/21 11:12:04 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:12:04 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:12:04 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:12:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:12:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:12:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56202->10.242.129.75:8301: i/o timeout
2021/05/21 11:12:06 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:12:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:12:09 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:12:09 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' joined, marking health alive
2021/05/21 11:12:10 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:12:10 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:12:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:12:12 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500174028s
2021/05/21 11:12:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56286->10.242.129.75:8301: i/o timeout
2021/05/21 11:12:12 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:12:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:12:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:40504->10.242.129.72:8301: i/o timeout
2021/05/21 11:12:14 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:12:15 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500302396s
2021/05/21 11:12:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:12:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:59288->10.242.129.75:8302: i/o timeout
2021/05/21 11:12:16 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:12:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:40516->10.242.129.72:8301: i/o timeout
2021/05/21 11:12:16 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:12:17 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.972325872s
2021/05/21 11:12:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:12:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56336->10.242.129.75:8301: i/o timeout
2021/05/21 11:12:18 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:12:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:12:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:40560->10.242.129.72:8301: i/o timeout
2021/05/21 11:12:20 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:12:22 [INFO] raft: aborting pipeline replication to peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}
2021/05/21 11:12:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56364->10.242.129.75:8301: i/o timeout
2021/05/21 11:12:22 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:12:22 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:12:23 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:12:23 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' failed, marking health critical
2021/05/21 11:12:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:12:24 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/21 11:12:24 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.75
2021/05/21 11:12:24 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:12:24 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/21 11:12:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:40590->10.242.129.72:8301: i/o timeout
2021/05/21 11:12:24 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:12:25 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500486918s
2021/05/21 11:12:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:12:26 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:12:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:59374->10.242.129.75:8302: i/o timeout
2021/05/21 11:12:26 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:12:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:12:28 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500333387s
2021/05/21 11:12:28 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:12:28 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:12:29 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:12:29 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.129.75:34038
2021/05/21 11:12:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:12:30 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.715907219s
2021/05/21 11:12:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:12:32 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-22bkc)
2021/05/21 11:12:32 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 7.144107336s
2021/05/21 11:12:32 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:50236->10.242.129.75:8300: i/o timeout
2021/05/21 11:12:32 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:12:33 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.75
2021/05/21 11:12:33 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:12:33 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/21 11:12:33 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:12:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:12:34 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:12:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56470->10.242.129.75:8301: i/o timeout
2021/05/21 11:12:35 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:12:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:12:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:12:37 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' joined, marking health alive
2021/05/21 11:12:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:12:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56518->10.242.129.75:8301: i/o timeout
2021/05/21 11:12:39 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:12:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:12:40 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500975864s
2021/05/21 11:12:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56540->10.242.129.75:8301: i/o timeout
2021/05/21 11:12:42 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:12:42 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:50310->10.242.129.75:8300: i/o timeout
2021/05/21 11:12:42 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:12:42 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:12:43 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/21 11:12:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:12:45 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-22bkc but other probes failed, network may be misconfigured
2021/05/21 11:12:46 [INFO] raft: pipelining replication to peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}
2021/05/21 11:12:46 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:12:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:12:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:12:50 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:12:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56614->10.242.129.75:8301: i/o timeout
2021/05/21 11:12:51 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:12:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:12:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:12:52 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500333337s
2021/05/21 11:12:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:12:54 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.741264925s
2021/05/21 11:12:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:40838->10.242.129.72:8301: i/o timeout
2021/05/21 11:12:54 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:12:54 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:12:54 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:12:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:12:56 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 6.52391399s
2021/05/21 11:12:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:59622->10.242.129.75:8302: i/o timeout
2021/05/21 11:12:56 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:12:56 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-22bkc but other probes failed, network may be misconfigured
2021/05/21 11:12:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56664->10.242.129.75:8301: i/o timeout
2021/05/21 11:12:57 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:12:58 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:12:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:12:58 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.50033699s
2021/05/21 11:12:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56674->10.242.129.75:8301: i/o timeout
2021/05/21 11:12:59 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:12:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:13:01 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-22bkc)
2021/05/21 11:13:04 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:13:04 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:13:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:40934->10.242.129.72:8301: i/o timeout
2021/05/21 11:13:05 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:13:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:13:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:13:08 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.5002644s
2021/05/21 11:13:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56750->10.242.129.75:8301: i/o timeout
2021/05/21 11:13:08 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:13:10 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:13:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:13:14 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:13:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:13:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:13:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:13:22 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:13:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:13:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:13:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:13:32 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:13:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:13:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:15:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:41988->10.242.129.72:8301: i/o timeout
2021/05/21 11:15:19 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:15:24 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:15:24 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:15:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:42058->10.242.129.72:8301: i/o timeout
2021/05/21 11:15:28 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:15:28 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:15:28 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:15:30 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:15:31 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/21 11:15:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:15:32 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500376023s
2021/05/21 11:15:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:57896->10.242.129.75:8301: i/o timeout
2021/05/21 11:15:33 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:15:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:15:34 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.948105719s
2021/05/21 11:15:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:42108->10.242.129.72:8301: i/o timeout
2021/05/21 11:15:35 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:15:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:15:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:42148->10.242.129.72:8301: i/o timeout
2021/05/21 11:15:37 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:15:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:15:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:57952->10.242.129.75:8301: i/o timeout
2021/05/21 11:15:39 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:15:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:15:40 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500227094s
2021/05/21 11:15:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:60950->10.242.129.75:8302: i/o timeout
2021/05/21 11:15:41 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:15:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:15:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:42198->10.242.129.72:8301: i/o timeout
2021/05/21 11:15:42 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:15:42 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.940194404s
2021/05/21 11:15:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:15:44 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/21 11:15:44 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.75
2021/05/21 11:15:44 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:15:44 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/21 11:15:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:58000->10.242.129.75:8301: i/o timeout
2021/05/21 11:15:44 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:15:44 [ERR] raft: Failed to pipeline AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: write tcp 10.242.1.135:50412->10.242.129.75:8300: use of closed network connection
2021/05/21 11:15:44 [INFO] raft: aborting pipeline replication to peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}
2021/05/21 11:15:44 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:15:44 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.75:8301
2021/05/21 11:15:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:15:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:42222->10.242.129.72:8301: i/o timeout
2021/05/21 11:15:46 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:15:47 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:15:47 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' failed, marking health critical
2021/05/21 11:15:47 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500268196s
2021/05/21 11:15:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:15:48 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:15:49 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:15:49 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/21 11:15:49 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.75
2021/05/21 11:15:49 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:15:49 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/21 11:15:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:15:50 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:15:50 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.129.72:36434
2021/05/21 11:15:51 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500232753s
2021/05/21 11:15:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:58064->10.242.129.75:8301: i/o timeout
2021/05/21 11:15:51 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:15:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:15:53 [INFO] raft: pipelining replication to peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}
2021/05/21 11:15:53 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/21 11:15:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:15:57 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:15:57 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' joined, marking health alive
2021/05/21 11:15:58 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:15:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:15:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:58126->10.242.129.75:8301: i/o timeout
2021/05/21 11:15:59 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:15:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:02 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-22bkc but other probes failed, network may be misconfigured
2021/05/21 11:16:02 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:16:02 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:16:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:42374->10.242.129.72:8301: i/o timeout
2021/05/21 11:16:03 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:16:03 [INFO] memberlist: Marking dev-consul-7467bf7b46-22bkc as failed, suspect timeout reached (2 peer confirmations)
2021/05/21 11:16:03 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:16:03 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' failed, marking health critical
2021/05/21 11:16:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:04 [ERR] memberlist: failed to receive: read tcp 10.242.1.135:8301->10.242.129.72:33466: i/o timeout from=10.242.129.72:33466
2021/05/21 11:16:04 [ERR] memberlist: Failed to send error: write tcp 10.242.1.135:8301->10.242.129.72:33466: i/o timeout from=10.242.129.72:33466
2021/05/21 11:16:04 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-22bkc)
2021/05/21 11:16:04 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:16:04 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' joined, marking health alive
2021/05/21 11:16:06 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:16:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:08 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500374124s
2021/05/21 11:16:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:42412->10.242.129.72:8301: i/o timeout
2021/05/21 11:16:08 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:16:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:12 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:16:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:16 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:16:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:42488->10.242.129.72:8301: i/o timeout
2021/05/21 11:16:17 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:16:17 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500192242s
2021/05/21 11:16:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:42502->10.242.129.72:8301: i/o timeout
2021/05/21 11:16:19 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:16:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:20 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.912666853s
2021/05/21 11:16:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:58304->10.242.129.75:8301: i/o timeout
2021/05/21 11:16:21 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:16:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:22 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 7.366949495s
2021/05/21 11:16:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:42534->10.242.129.72:8301: i/o timeout
2021/05/21 11:16:23 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:16:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:25 [INFO] raft: aborting pipeline replication to peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}
2021/05/21 11:16:25 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:49810->10.242.129.75:8300: i/o timeout
2021/05/21 11:16:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:58338->10.242.129.75:8301: i/o timeout
2021/05/21 11:16:25 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:16:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:33094->10.242.129.75:8302: i/o timeout
2021/05/21 11:16:26 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:16:27 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500380705s
2021/05/21 11:16:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:28 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.75
2021/05/21 11:16:28 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:16:28 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/21 11:16:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:42582->10.242.129.72:8301: i/o timeout
2021/05/21 11:16:28 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:16:29 [INFO] memberlist: Marking dev-consul-7467bf7b46-22bkc as failed, suspect timeout reached (1 peer confirmations)
2021/05/21 11:16:29 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:16:29 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' failed, marking health critical
2021/05/21 11:16:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:30 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.964223642s
2021/05/21 11:16:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:32 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 7.448000264s
2021/05/21 11:16:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:35 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:52212->10.242.129.75:8300: i/o timeout
2021/05/21 11:16:35 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:16:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:35 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:52220->10.242.129.75:8300: i/o timeout
2021/05/21 11:16:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:33140->10.242.129.75:8302: i/o timeout
2021/05/21 11:16:36 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:16:36 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:16:37 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:16:37 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.129.75:37000
2021/05/21 11:16:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:42 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:16:42 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:16:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:45 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:52300->10.242.129.75:8300: i/o timeout
2021/05/21 11:16:45 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:16:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:33272->10.242.129.75:8302: i/o timeout
2021/05/21 11:16:46 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:16:46 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:52320->10.242.129.75:8300: i/o timeout
2021/05/21 11:16:47 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:16:47 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.75:8301
2021/05/21 11:16:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:50 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/21 11:16:50 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.129.75
2021/05/21 11:16:50 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/21 11:16:51 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.129.75:8302
2021/05/21 11:16:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:55 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:52390->10.242.129.75:8300: i/o timeout
2021/05/21 11:16:55 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:16:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:57 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:52402->10.242.129.75:8300: i/o timeout
2021/05/21 11:16:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:57 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:16:59 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/21 11:16:59 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.129.75
2021/05/21 11:16:59 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/21 11:16:59 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.75
2021/05/21 11:16:59 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:16:59 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/21 11:16:59 [WARN] raft: Rejecting vote request from 10.242.129.75:8300 since we have a leader: 10.242.1.135:8300
2021/05/21 11:16:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:16:59 [ERR] raft: peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300} has newer term, stopping replication
2021/05/21 11:16:59 [INFO] raft: Node at 10.242.1.135:8300 [Follower] entering Follower state (Leader: "")
2021/05/21 11:16:59 [INFO] raft: aborting pipeline replication to peer {Voter c955779f-89a6-203f-a38f-22883c86e868 10.242.142.201:8300}
2021/05/21 11:16:59 [INFO] consul: cluster leadership lost
2021/05/21 11:17:00 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:17:06 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/05/21 11:17:06 [INFO] raft: Node at 10.242.1.135:8300 [Candidate] entering Candidate state in term 6
2021/05/21 11:17:07 [INFO] raft: Election won. Tally: 2
2021/05/21 11:17:07 [INFO] raft: Node at 10.242.1.135:8300 [Leader] entering Leader state
2021/05/21 11:17:07 [INFO] raft: Added peer 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, starting replication
2021/05/21 11:17:07 [INFO] raft: Added peer c955779f-89a6-203f-a38f-22883c86e868, starting replication
2021/05/21 11:17:07 [INFO] consul: cluster leadership acquired
2021/05/21 11:17:07 [INFO] consul: New leader elected: dev-consul-server-0
2021/05/21 11:17:07 [INFO] raft: pipelining replication to peer {Voter c955779f-89a6-203f-a38f-22883c86e868 10.242.142.201:8300}
2021/05/21 11:17:07 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' joined, marking health alive
2021/05/21 11:17:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:42914->10.242.129.72:8301: i/o timeout
2021/05/21 11:17:07 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:17:09 [WARN] raft: AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300} rejected, sending older logs (next: 572)
2021/05/21 11:17:09 [INFO] raft: pipelining replication to peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}
2021/05/21 11:17:16 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:17:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:58792->10.242.129.75:8301: i/o timeout
2021/05/21 11:17:18 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:17:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:58806->10.242.129.75:8301: i/o timeout
2021/05/21 11:17:18 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:17:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:17:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:17:21 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-22bkc but other probes failed, network may be misconfigured
2021/05/21 11:17:21 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.75
2021/05/21 11:17:21 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:17:21 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/21 11:17:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:17:23 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.50024937s
2021/05/21 11:17:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:43058->10.242.129.72:8301: i/o timeout
2021/05/21 11:17:23 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:17:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:17:25 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:17:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:43078->10.242.129.72:8301: i/o timeout
2021/05/21 11:17:25 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:17:26 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:17:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:33616->10.242.129.75:8302: i/o timeout
2021/05/21 11:17:26 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:17:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:17:27 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.75
2021/05/21 11:17:27 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:17:27 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/21 11:17:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:43094->10.242.129.72:8301: i/o timeout
2021/05/21 11:17:27 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:17:27 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/21 11:17:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:58924->10.242.129.75:8301: i/o timeout
2021/05/21 11:17:29 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:17:29 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500261242s
2021/05/21 11:17:30 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:17:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:17:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:43144->10.242.129.72:8301: i/o timeout
2021/05/21 11:17:31 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:17:32 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:17:32 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' failed, marking health critical
2021/05/21 11:17:32 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.982704937s
2021/05/21 11:17:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:17:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:58964->10.242.129.75:8301: i/o timeout
2021/05/21 11:17:33 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:17:34 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 7.353627852s
2021/05/21 11:17:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:17:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:58978->10.242.129.75:8301: i/o timeout
2021/05/21 11:17:35 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:17:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:17:37 [INFO] raft: aborting pipeline replication to peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}
2021/05/21 11:17:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:59006->10.242.129.75:8301: i/o timeout
2021/05/21 11:17:37 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:17:37 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:52470->10.242.129.75:8300: i/o timeout
2021/05/21 11:17:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:17:39 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500279815s
2021/05/21 11:17:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:59038->10.242.129.75:8301: i/o timeout
2021/05/21 11:17:40 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:17:41 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/21 11:17:41 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.75
2021/05/21 11:17:41 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:17:41 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/21 11:17:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:17:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:33784->10.242.129.75:8302: i/o timeout
2021/05/21 11:17:41 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:17:42 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.952038309s
2021/05/21 11:17:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:59050->10.242.129.75:8301: i/o timeout
2021/05/21 11:17:42 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:17:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:17:44 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 7.413320127s
2021/05/21 11:17:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:17:46 [ERR] memberlist: failed to receive: read tcp 10.242.1.135:8301->10.242.129.75:52382: i/o timeout from=10.242.129.75:52382
2021/05/21 11:17:46 [ERR] memberlist: Failed to send error: write tcp 10.242.1.135:8301->10.242.129.75:52382: i/o timeout from=10.242.129.75:52382
2021/05/21 11:17:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:17:47 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:52574->10.242.129.75:8300: i/o timeout
2021/05/21 11:17:47 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:17:48 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:52874->10.242.129.75:8300: i/o timeout
2021/05/21 11:17:49 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:17:49 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.129.72:40776
2021/05/21 11:17:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:17:49 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:17:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:17:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:17:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:33826->10.242.129.75:8302: i/o timeout
2021/05/21 11:17:51 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:17:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:17:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:17:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:17:57 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:52952->10.242.129.75:8300: i/o timeout
2021/05/21 11:17:57 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:17:57 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.75:8301
2021/05/21 11:17:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:17:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:17:59 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:52970->10.242.129.75:8300: i/o timeout
2021/05/21 11:17:59 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:18:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:33954->10.242.129.75:8302: i/o timeout
2021/05/21 11:18:01 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:18:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:06 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:18:06 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:18:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:18:07 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:53038->10.242.129.75:8300: i/o timeout
2021/05/21 11:18:07 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:18:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:09 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:53064->10.242.129.75:8300: i/o timeout
2021/05/21 11:18:10 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:18:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:11 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/21 11:18:11 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.129.75
2021/05/21 11:18:11 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/21 11:18:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:17 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:53126->10.242.129.75:8300: i/o timeout
2021/05/21 11:18:17 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:18:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:20 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:53156->10.242.129.75:8300: i/o timeout
2021/05/21 11:18:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:21 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:18:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:25 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.75
2021/05/21 11:18:25 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:18:25 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/21 11:18:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:59432->10.242.129.75:8301: i/o timeout
2021/05/21 11:18:26 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:18:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:27 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:53220->10.242.129.75:8300: i/o timeout
2021/05/21 11:18:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:59446->10.242.129.75:8301: i/o timeout
2021/05/21 11:18:28 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:18:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:29 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.129.75
2021/05/21 11:18:29 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/21 11:18:30 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/21 11:18:30 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.75
2021/05/21 11:18:30 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:18:30 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/21 11:18:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:31 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:18:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:59476->10.242.129.75:8301: i/o timeout
2021/05/21 11:18:31 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:18:32 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:18:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:33 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:18:34 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:18:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:35 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:18:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:34252->10.242.129.75:8302: i/o timeout
2021/05/21 11:18:36 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:18:36 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500194381s
2021/05/21 11:18:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:37 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:53298->10.242.129.75:8300: i/o timeout
2021/05/21 11:18:37 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:18:37 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.75:8301
2021/05/21 11:18:38 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/21 11:18:38 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:18:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:40 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500276761s
2021/05/21 11:18:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:43 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.966211127s
2021/05/21 11:18:44 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:18:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:45 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:18:46 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:18:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:34318->10.242.129.75:8302: i/o timeout
2021/05/21 11:18:46 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:18:46 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:18:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:47 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:53404->10.242.129.75:8300: i/o timeout
2021/05/21 11:18:47 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:18:48 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:18:48 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:18:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:50 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500307274s
2021/05/21 11:18:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:52 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.939447212s
2021/05/21 11:18:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:55 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 7.419343774s
2021/05/21 11:18:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:58 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:53256->10.242.129.75:8300: i/o timeout
2021/05/21 11:18:58 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:53490->10.242.129.75:8300: i/o timeout
2021/05/21 11:18:58 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:18:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:18:59 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:19:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:34476->10.242.129.75:8302: i/o timeout
2021/05/21 11:19:01 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:19:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:06 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/21 11:19:06 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.129.75
2021/05/21 11:19:06 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/21 11:19:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:07 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:19:07 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.129.75:40368
2021/05/21 11:19:07 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:19:07 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/21 11:19:08 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:53570->10.242.129.75:8300: i/o timeout
2021/05/21 11:19:09 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:53576->10.242.129.75:8300: i/o timeout
2021/05/21 11:19:09 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:19:10 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:19:10 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:19:10 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:19:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:34560->10.242.129.75:8302: i/o timeout
2021/05/21 11:19:11 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:19:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:17 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.75:8301
2021/05/21 11:19:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:19 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:53670->10.242.129.75:8300: i/o timeout
2021/05/21 11:19:20 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:53674->10.242.129.75:8300: i/o timeout
2021/05/21 11:19:20 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:19:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:22 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:19:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:30 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:53782->10.242.129.75:8300: i/o timeout
2021/05/21 11:19:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:31 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:19:32 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:19:32 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:19:32 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:53808->10.242.129.75:8300: i/o timeout
2021/05/21 11:19:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:19:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:37 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:19:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:41 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:53880->10.242.129.75:8300: i/o timeout
2021/05/21 11:19:42 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:19:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:47 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:53942->10.242.129.75:8300: i/o timeout
2021/05/21 11:19:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:49 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:19:49 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/21 11:19:52 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:19:52 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:19:52 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:53986->10.242.129.75:8300: i/o timeout
2021/05/21 11:19:52 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:19:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:19:57 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.75:8301
2021/05/21 11:19:58 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:19:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:02 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:54080->10.242.129.75:8300: i/o timeout
2021/05/21 11:20:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:04 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:20:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:20:08 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:54120->10.242.129.75:8300: i/o timeout
2021/05/21 11:20:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:14 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:54174->10.242.129.75:8300: i/o timeout
2021/05/21 11:20:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:15 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:20:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:18 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:20:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:21 [ERR] memberlist: failed to receive: read tcp 10.242.1.135:8302->10.242.129.75:48038: i/o timeout from=10.242.129.75:48038
2021/05/21 11:20:21 [ERR] memberlist: Failed to send error: write tcp 10.242.1.135:8302->10.242.129.75:48038: i/o timeout from=10.242.129.75:48038
2021/05/21 11:20:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:25 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:54276->10.242.129.75:8300: i/o timeout
2021/05/21 11:20:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:27 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:20:28 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:54300->10.242.129.75:8300: i/o timeout
2021/05/21 11:20:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:31 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:20:31 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/21 11:20:34 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:20:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:37 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:54378->10.242.129.75:8300: i/o timeout
2021/05/21 11:20:37 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.75:8301
2021/05/21 11:20:38 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:20:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:40 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:20:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:46 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:20:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:48 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:54488->10.242.129.75:8300: i/o timeout
2021/05/21 11:20:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:50 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:54514->10.242.129.75:8300: i/o timeout
2021/05/21 11:20:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:53 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:20:53 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' joined, marking health alive
2021/05/21 11:20:54 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.75
2021/05/21 11:20:54 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:20:54 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/21 11:20:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:44978->10.242.129.72:8301: i/o timeout
2021/05/21 11:20:55 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:20:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:20:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:45008->10.242.129.72:8301: i/o timeout
2021/05/21 11:20:58 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:20:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:21:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:60812->10.242.129.75:8301: i/o timeout
2021/05/21 11:21:00 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:21:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:21:01 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.129.75
2021/05/21 11:21:01 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/21 11:21:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:45052->10.242.129.72:8301: i/o timeout
2021/05/21 11:21:02 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:21:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:21:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:60860->10.242.129.75:8301: i/o timeout
2021/05/21 11:21:04 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:21:05 [INFO] raft: pipelining replication to peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}
2021/05/21 11:21:06 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/21 11:21:08 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:21:12 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:21:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:21:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:21:14 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500258598s
2021/05/21 11:21:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:60948->10.242.129.75:8301: i/o timeout
2021/05/21 11:21:14 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:21:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:21:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:60958->10.242.129.75:8301: i/o timeout
2021/05/21 11:21:16 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:21:16 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.970367174s
2021/05/21 11:21:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:21:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:45188->10.242.129.72:8301: i/o timeout
2021/05/21 11:21:18 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:21:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:21:19 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 7.416161904s
2021/05/21 11:21:20 [ERR] raft: Failed to pipeline AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: write tcp 10.242.1.135:54662->10.242.129.75:8300: use of closed network connection
2021/05/21 11:21:20 [INFO] raft: aborting pipeline replication to peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}
2021/05/21 11:21:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:45210->10.242.129.72:8301: i/o timeout
2021/05/21 11:21:20 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:21:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:32798->10.242.129.75:8301: i/o timeout
2021/05/21 11:21:22 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:21:22 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:54648->10.242.129.75:8300: i/o timeout
2021/05/21 11:21:23 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500287932s
2021/05/21 11:21:24 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:21:24 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:21:24 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:21:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:32816->10.242.129.75:8301: i/o timeout
2021/05/21 11:21:24 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:21:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:21:25 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.953159572s
2021/05/21 11:21:26 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/21 11:21:26 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.75
2021/05/21 11:21:26 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:21:26 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/21 11:21:26 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:21:26 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' failed, marking health critical
2021/05/21 11:21:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:45266->10.242.129.72:8301: i/o timeout
2021/05/21 11:21:26 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:21:26 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:21:26 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.129.75:41614
2021/05/21 11:21:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:21:28 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 7.428831448s
2021/05/21 11:21:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:21:30 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:54870->10.242.129.75:8300: i/o timeout
2021/05/21 11:21:30 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:21:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:21:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:21:33 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:54904->10.242.129.75:8300: i/o timeout
2021/05/21 11:21:34 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:21:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:21:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:21:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:21:40 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:54960->10.242.129.75:8300: i/o timeout
2021/05/21 11:21:40 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:21:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:21:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:35948->10.242.129.75:8302: i/o timeout
2021/05/21 11:21:41 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:21:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:21:43 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:21:43 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/21 11:21:44 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:54996->10.242.129.75:8300: i/o timeout
2021/05/21 11:21:45 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:21:46 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:21:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:21:47 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.75:8301
2021/05/21 11:21:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:21:50 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:55070->10.242.129.75:8300: i/o timeout
2021/05/21 11:21:50 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:21:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:21:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:36038->10.242.129.75:8302: i/o timeout
2021/05/21 11:21:51 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:21:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:21:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:21:55 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:55112->10.242.129.75:8300: i/o timeout
2021/05/21 11:21:55 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:21:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:21:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:00 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:55158->10.242.129.75:8300: i/o timeout
2021/05/21 11:22:00 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:22:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:36128->10.242.129.75:8302: i/o timeout
2021/05/21 11:22:01 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:22:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:05 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/21 11:22:05 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.129.75
2021/05/21 11:22:05 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/21 11:22:05 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:55208->10.242.129.75:8300: i/o timeout
2021/05/21 11:22:06 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:22:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:10 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:55250->10.242.129.75:8300: i/o timeout
2021/05/21 11:22:10 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:22:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:36174->10.242.129.75:8302: i/o timeout
2021/05/21 11:22:11 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:22:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:16 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:55304->10.242.129.75:8300: i/o timeout
2021/05/21 11:22:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:17 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:22:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:20 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:55342->10.242.129.75:8300: i/o timeout
2021/05/21 11:22:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:21 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:22:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:25 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:22:25 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/21 11:22:27 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:55400->10.242.129.75:8300: i/o timeout
2021/05/21 11:22:27 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.75:8301
2021/05/21 11:22:28 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:22:28 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:22:28 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.129.75:8302
2021/05/21 11:22:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:30 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:22:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:31 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:55432->10.242.129.75:8300: i/o timeout
2021/05/21 11:22:31 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:22:31 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:22:31 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:22:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:33 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500324653s
2021/05/21 11:22:34 [WARN] raft: Rejecting vote request from 10.242.129.75:8300 since we have a leader: 10.242.1.135:8300
2021/05/21 11:22:34 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.129.75
2021/05/21 11:22:34 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/21 11:22:35 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/21 11:22:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:35 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:22:36 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:22:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:38 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:22:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:39 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.75
2021/05/21 11:22:39 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:22:39 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/21 11:22:39 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500188776s
2021/05/21 11:22:41 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:55490->10.242.129.75:8300: i/o timeout
2021/05/21 11:22:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:33538->10.242.129.75:8301: i/o timeout
2021/05/21 11:22:41 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:22:42 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:22:42 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.991129047s
2021/05/21 11:22:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:33550->10.242.129.75:8301: i/o timeout
2021/05/21 11:22:43 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:22:44 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 7.443793143s
2021/05/21 11:22:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:33572->10.242.129.75:8301: i/o timeout
2021/05/21 11:22:45 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:22:45 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/21 11:22:45 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.75
2021/05/21 11:22:45 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:22:45 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/21 11:22:46 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/21 11:22:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:48 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:55528->10.242.129.75:8300: i/o timeout
2021/05/21 11:22:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:49 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:22:50 [ERR] memberlist: Push/Pull with dev-consul-server-1 failed: read tcp 10.242.1.135:33524->10.242.129.75:8301: i/o timeout
2021/05/21 11:22:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:36594->10.242.129.75:8302: i/o timeout
2021/05/21 11:22:51 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:22:52 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:55638->10.242.129.75:8300: i/o timeout
2021/05/21 11:22:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:53 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:22:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:22:59 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:55692->10.242.129.75:8300: i/o timeout
2021/05/21 11:22:59 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:23:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:23:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:23:03 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:55738->10.242.129.75:8300: i/o timeout
2021/05/21 11:23:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:23:05 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:23:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:36734->10.242.129.75:8302: i/o timeout
2021/05/21 11:23:06 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:23:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:23:07 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:23:07 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/21 11:23:07 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-22bkc 10.242.129.72:8301
2021/05/21 11:23:09 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:55790->10.242.129.75:8300: i/o timeout
2021/05/21 11:23:09 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-z9mlm 10.242.1.149
2021/05/21 11:23:09 [INFO] consul: member 'dev-consul-7467bf7b46-z9mlm' joined, marking health alive
2021/05/21 11:23:10 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:23:10 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:23:10 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:23:10 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.129.75:42222
2021/05/21 11:23:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:23:12 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:23:12 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' joined, marking health alive
2021/05/21 11:23:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:23:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:46370->10.242.129.72:8301: i/o timeout
2021/05/21 11:23:14 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:23:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:23:15 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:55860->10.242.129.75:8300: i/o timeout
2021/05/21 11:23:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:36920->10.242.129.75:8302: i/o timeout
2021/05/21 11:23:16 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:23:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:23:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:46436->10.242.129.72:8301: i/o timeout
2021/05/21 11:23:18 [INFO] memberlist: Marking dev-consul-7467bf7b46-22bkc as failed, suspect timeout reached (2 peer confirmations)
2021/05/21 11:23:18 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:23:18 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:23:18 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' failed, marking health critical
2021/05/21 11:23:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:23:20 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:55922->10.242.129.75:8300: i/o timeout
2021/05/21 11:23:20 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/21 11:23:20 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.129.75
2021/05/21 11:23:20 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/21 11:23:21 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:23:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:23:21 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:23:22 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:23:22 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.129.72:42348
2021/05/21 11:23:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:23:24 [ERR] memberlist: failed to receive: read tcp 10.242.1.135:8301->10.242.129.75:54002: i/o timeout from=10.242.129.75:54002
2021/05/21 11:23:24 [ERR] memberlist: Failed to send error: write tcp 10.242.1.135:8301->10.242.129.75:54002: i/o timeout from=10.242.129.75:54002
2021/05/21 11:23:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:23:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:36996->10.242.129.75:8302: i/o timeout
2021/05/21 11:23:26 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:23:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:23:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:23:31 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:56134->10.242.129.75:8300: i/o timeout
2021/05/21 11:23:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:23:31 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: read tcp 10.242.1.135:56144->10.242.129.75:8300: i/o timeout
2021/05/21 11:23:31 [WARN] memberlist: Refuting a dead message (from: dev-consul-server-1.dc1)
2021/05/21 11:23:32 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:23:32 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-22bkc)
2021/05/21 11:23:32 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:23:32 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' joined, marking health alive
2021/05/21 11:23:33 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:23:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:23:33 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:23:33 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.129.75
2021/05/21 11:23:33 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/21 11:23:33 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.75
2021/05/21 11:23:33 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:23:34 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:23:34 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/21 11:23:36 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:23:36 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:23:40 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:23:40 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:23:42 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:23:42 [INFO] raft: pipelining replication to peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}
2021/05/21 11:23:44 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:23:44 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:23:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:23:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:23:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:34480->10.242.129.75:8301: i/o timeout
2021/05/21 11:23:46 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:23:46 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500205129s
2021/05/21 11:23:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:23:48 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:23:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:34604->10.242.129.75:8301: i/o timeout
2021/05/21 11:23:53 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:23:55 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/21 11:24:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:47132->10.242.129.72:8301: i/o timeout
2021/05/21 11:24:00 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:24:01 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/21 11:24:04 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:24:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:24:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:34780->10.242.129.75:8301: i/o timeout
2021/05/21 11:24:05 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:24:12 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:24:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:24:12 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:24:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:47324->10.242.129.72:8301: i/o timeout
2021/05/21 11:24:12 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:24:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:24:16 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:24:16 [INFO] memberlist: Marking dev-consul-7467bf7b46-22bkc as failed, suspect timeout reached (2 peer confirmations)
2021/05/21 11:24:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:34902->10.242.129.75:8301: i/o timeout
2021/05/21 11:24:16 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:24:16 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:24:16 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' failed, marking health critical
2021/05/21 11:24:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:24:17 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-22bkc 10.242.129.72:8301
2021/05/21 11:24:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:24:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:24:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:35028->10.242.129.75:8301: i/o timeout
2021/05/21 11:24:21 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:24:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:24:23 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/21 11:24:23 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.75
2021/05/21 11:24:23 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:24:23 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/21 11:24:24 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:24:24 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/21 11:24:25 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:24:26 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:24:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:24:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:24:27 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:24:27 [INFO] raft: aborting pipeline replication to peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}
2021/05/21 11:24:27 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:24:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:24:29 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.75
2021/05/21 11:24:29 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:24:29 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/21 11:24:30 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:24:30 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' joined, marking health alive
2021/05/21 11:24:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:24:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:47638->10.242.129.72:8301: i/o timeout
2021/05/21 11:24:32 [INFO] memberlist: Suspect dev-consul-7467bf7b46-22bkc has failed, no acks received
2021/05/21 11:24:32 [INFO] raft: pipelining replication to peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}
2021/05/21 11:24:44 [INFO] serf: EventMemberLeave: dev-consul-7467bf7b46-22bkc 10.242.129.72
2021/05/21 11:24:44 [INFO] consul: member 'dev-consul-7467bf7b46-22bkc' left, deregistering
2021/05/21 11:24:44 [INFO] raft: aborting pipeline replication to peer {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}
2021/05/21 11:24:44 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: EOF
2021/05/21 11:24:45 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:45 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/21 11:24:45 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: read tcp 10.242.1.135:57530->10.242.129.75:8300: read: connection reset by peer
2021/05/21 11:24:45 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:45 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:45 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:46 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:46 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:24:46 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:46 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:46 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:46 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:47 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error getting client: failed to get conn: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:47 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:47 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 2.500395592s
2021/05/21 11:24:47 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:48 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:48 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:24:48 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:24:48 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:24:48 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:49 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:49 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error getting client: failed to get conn: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:49 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 4.950650323s
2021/05/21 11:24:49 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:49 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/21 11:24:49 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.75
2021/05/21 11:24:49 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.75:8300) (DC: dc1)
2021/05/21 11:24:49 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/21 11:24:50 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:24:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:24:50 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:24:51 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error getting client: failed to get conn: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:51 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:24:51 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:51 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:24:51 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:52 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:24:52 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:24:52 [WARN] raft: Failed to contact 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be in 7.401021505s
2021/05/21 11:24:52 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:24:52 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:53 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error getting client: failed to get conn: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:54 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:24:54 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:24:54 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:24:54 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:24:54 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:55 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error getting client: failed to get conn: dial tcp ->10.242.129.75:8300: getsockopt: connection refused
2021/05/21 11:24:56 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:24:56 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:24:56 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:24:58 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:24:58 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:24:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:24:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:06 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: dial tcp ->10.242.129.75:8300: i/o timeout
2021/05/21 11:25:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:07 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error getting client: failed to get conn: dial tcp ->10.242.129.75:8300: i/o timeout
2021/05/21 11:25:08 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: dial tcp ->10.242.129.75:8300: i/o timeout
2021/05/21 11:25:10 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:25:10 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:25:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:13 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:25:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:15 [ERR] memberlist: Conflicting address for dev-consul-server-1. Mine: 10.242.129.75:8301 Theirs: 10.242.129.99:8301
2021/05/21 11:25:15 [WARN] serf: Name conflict for 'dev-consul-server-1' both 10.242.129.75:8301 and 10.242.129.99:8301 are claiming
2021/05/21 11:25:15 [ERR] memberlist: Conflicting address for dev-consul-server-1.dc1. Mine: 10.242.129.75:8302 Theirs: 10.242.129.99:8302
2021/05/21 11:25:15 [WARN] serf: Name conflict for 'dev-consul-server-1.dc1' both 10.242.129.75:8302 and 10.242.129.99:8302 are claiming
2021/05/21 11:25:15 [ERR] memberlist: Conflicting address for dev-consul-server-1. Mine: 10.242.129.75:8301 Theirs: 10.242.129.99:8301
2021/05/21 11:25:15 [WARN] serf: Name conflict for 'dev-consul-server-1' both 10.242.129.75:8301 and 10.242.129.99:8301 are claiming
2021/05/21 11:25:15 [ERR] memberlist: Conflicting address for dev-consul-server-1. Mine: 10.242.129.75:8301 Theirs: 10.242.129.99:8301
2021/05/21 11:25:15 [WARN] serf: Name conflict for 'dev-consul-server-1' both 10.242.129.75:8301 and 10.242.129.99:8301 are claiming
2021/05/21 11:25:16 [ERR] memberlist: Conflicting address for dev-consul-server-1.dc1. Mine: 10.242.129.75:8302 Theirs: 10.242.129.99:8302
2021/05/21 11:25:16 [WARN] serf: Name conflict for 'dev-consul-server-1.dc1' both 10.242.129.75:8302 and 10.242.129.99:8302 are claiming
2021/05/21 11:25:16 [ERR] memberlist: Conflicting address for dev-consul-server-1.dc1. Mine: 10.242.129.75:8302 Theirs: 10.242.129.99:8302
2021/05/21 11:25:16 [WARN] serf: Name conflict for 'dev-consul-server-1.dc1' both 10.242.129.75:8302 and 10.242.129.99:8302 are claiming
2021/05/21 11:25:16 [ERR] memberlist: Conflicting address for dev-consul-server-1.dc1. Mine: 10.242.129.75:8302 Theirs: 10.242.129.99:8302
2021/05/21 11:25:16 [WARN] serf: Name conflict for 'dev-consul-server-1.dc1' both 10.242.129.75:8302 and 10.242.129.99:8302 are claiming
2021/05/21 11:25:16 [ERR] memberlist: Conflicting address for dev-consul-server-1.dc1. Mine: 10.242.129.75:8302 Theirs: 10.242.129.99:8302
2021/05/21 11:25:16 [WARN] serf: Name conflict for 'dev-consul-server-1.dc1' both 10.242.129.75:8302 and 10.242.129.99:8302 are claiming
2021/05/21 11:25:17 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:25:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:19 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error getting client: failed to get conn: dial tcp ->10.242.129.75:8300: i/o timeout
2021/05/21 11:25:22 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:25:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:23 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: dial tcp ->10.242.129.75:8300: i/o timeout
2021/05/21 11:25:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:26 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.129.75
2021/05/21 11:25:26 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/21 11:25:27 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: dial tcp ->10.242.129.75:8300: i/o timeout
2021/05/21 11:25:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:29 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:25:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:31 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error getting client: failed to get conn: dial tcp ->10.242.129.75:8300: i/o timeout
2021/05/21 11:25:34 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:25:34 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:25:35 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:25:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:37 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:25:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:43 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error getting client: failed to get conn: dial tcp ->10.242.129.75:8300: i/o timeout
2021/05/21 11:25:45 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: dial tcp ->10.242.129.75:8300: i/o timeout
2021/05/21 11:25:46 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:25:46 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:25:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:47 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: dial tcp ->10.242.129.75:8300: i/o timeout
2021/05/21 11:25:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:25:55 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error getting client: failed to get conn: dial tcp ->10.242.129.75:8300: i/o timeout
2021/05/21 11:25:56 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:25:57 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:25:58 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:25:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:26:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:26:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:26:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:26:06 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: dial tcp ->10.242.129.75:8300: i/o timeout
2021/05/21 11:26:07 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error getting client: failed to get conn: dial tcp ->10.242.129.75:8300: i/o timeout
2021/05/21 11:26:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:26:07 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.99
2021/05/21 11:26:07 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.99:8300) (DC: dc1)
2021/05/21 11:26:07 [INFO] raft: Updating configuration with AddNonvoter (7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, 10.242.129.99:8300) to [{Suffrage:Voter ID:9959677e-074a-a336-5a52-0f08e5968308 Address:10.242.1.135:8300} {Suffrage:Voter ID:1ab9e202-faa9-c71c-bdae-e0e5ae74e4be Address:10.242.129.75:8300} {Suffrage:Voter ID:c955779f-89a6-203f-a38f-22883c86e868 Address:10.242.142.201:8300} {Suffrage:Nonvoter ID:7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca Address:10.242.129.99:8300}]
2021/05/21 11:26:07 [INFO] raft: Added peer 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, starting replication
2021/05/21 11:26:07 [WARN] raft: AppendEntries to {Nonvoter 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca 10.242.129.99:8300} rejected, sending older logs (next: 1)
2021/05/21 11:26:07 [INFO] raft: pipelining replication to peer {Nonvoter 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca 10.242.129.99:8300}
2021/05/21 11:26:07 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: dial tcp ->10.242.129.75:8300: i/o timeout
2021/05/21 11:26:07 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/21 11:26:17 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:26:17 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:26:27 [INFO] autopilot: Promoting Server (ID: "7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca" Address: "10.242.129.99:8300") to voter
2021/05/21 11:26:27 [INFO] raft: Updating configuration with AddStaging (7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, 10.242.129.99:8300) to [{Suffrage:Voter ID:9959677e-074a-a336-5a52-0f08e5968308 Address:10.242.1.135:8300} {Suffrage:Voter ID:1ab9e202-faa9-c71c-bdae-e0e5ae74e4be Address:10.242.129.75:8300} {Suffrage:Voter ID:c955779f-89a6-203f-a38f-22883c86e868 Address:10.242.142.201:8300} {Suffrage:Voter ID:7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca Address:10.242.129.99:8300}]
2021/05/21 11:26:27 [INFO] autopilot: Attempting removal of stale Server (ID: "1ab9e202-faa9-c71c-bdae-e0e5ae74e4be" Address: "10.242.129.75:8300")
2021/05/21 11:26:27 [INFO] raft: Updating configuration with RemoveServer (1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, ) to [{Suffrage:Voter ID:9959677e-074a-a336-5a52-0f08e5968308 Address:10.242.1.135:8300} {Suffrage:Voter ID:c955779f-89a6-203f-a38f-22883c86e868 Address:10.242.142.201:8300} {Suffrage:Voter ID:7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca Address:10.242.129.99:8300}]
2021/05/21 11:26:27 [INFO] raft: Removed peer 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, stopping replication after 836
2021/05/21 11:26:27 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: dial tcp ->10.242.129.75:8300: i/o timeout
2021/05/21 11:26:27 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: dial tcp ->10.242.129.75:8300: i/o timeout
2021/05/21 11:26:35 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.129.75:8302
2021/05/21 11:26:37 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:26:38 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:26:47 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: dial tcp ->10.242.129.75:8300: i/o timeout
2021/05/21 11:26:48 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: dial tcp ->10.242.129.75:8300: i/o timeout
2021/05/21 11:26:58 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:26:59 [WARN] Unable to get address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be, using fallback address 10.242.129.75:8300: Could not find address for server id 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be
2021/05/21 11:27:00 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.129.99
2021/05/21 11:27:00 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/21 11:27:08 [ERR] raft: Failed to AppendEntries to {Voter 1ab9e202-faa9-c71c-bdae-e0e5ae74e4be 10.242.129.75:8300}: dial tcp ->10.242.129.75:8300: i/o timeout
2021/05/21 11:27:09 [ERR] raft: Failed to heartbeat to 10.242.129.75:8300: dial tcp ->10.242.129.75:8300: i/o timeout
2021/05/21 11:41:42 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:41:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/21 11:44:00 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:44:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:47968->10.242.129.99:8301: i/o timeout
2021/05/21 11:44:01 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:44:02 [WARN] raft: Failed to contact 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca in 2.500297465s
2021/05/21 11:44:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:48014->10.242.129.99:8301: i/o timeout
2021/05/21 11:44:04 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:44:04 [WARN] raft: Failed to contact 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca in 4.965417958s
2021/05/21 11:44:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:05 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/21 11:44:05 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.99
2021/05/21 11:44:05 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.99:8300) (DC: dc1)
2021/05/21 11:44:05 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/21 11:44:05 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:44:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:07 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:44:08 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.99
2021/05/21 11:44:08 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.99:8300) (DC: dc1)
2021/05/21 11:44:08 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/21 11:44:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:09 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/21 11:44:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:48166->10.242.129.99:8301: i/o timeout
2021/05/21 11:44:11 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:44:12 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:44:12 [WARN] raft: Failed to contact 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca in 2.500257493s
2021/05/21 11:44:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:48210->10.242.129.99:8301: i/o timeout
2021/05/21 11:44:14 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:44:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:15 [WARN] raft: Failed to contact 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca in 4.998845116s
2021/05/21 11:44:15 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/21 11:44:15 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.99
2021/05/21 11:44:15 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.99:8300) (DC: dc1)
2021/05/21 11:44:15 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/21 11:44:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:17 [WARN] raft: Failed to contact 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca in 7.477386288s
2021/05/21 11:44:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:19 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:44:19 [INFO] raft: aborting pipeline replication to peer {Nonvoter 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca 10.242.129.99:8300}
2021/05/21 11:44:19 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:44:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:48144->10.242.129.99:8302: i/o timeout
2021/05/21 11:44:21 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:44:21 [WARN] raft: Failed to contact 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca in 2.500361381s
2021/05/21 11:44:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:24 [WARN] raft: Failed to contact 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca in 4.943242828s
2021/05/21 11:44:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:26 [WARN] raft: Failed to contact 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca in 7.421116546s
2021/05/21 11:44:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:29 [ERR] raft: Failed to heartbeat to 10.242.129.99:8300: read tcp 10.242.1.135:37974->10.242.129.99:8300: i/o timeout
2021/05/21 11:44:29 [ERR] raft: Failed to AppendEntries to {Nonvoter 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca 10.242.129.99:8300}: read tcp 10.242.1.135:57888->10.242.129.99:8300: i/o timeout
2021/05/21 11:44:29 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:44:29 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:44:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:39 [ERR] raft: Failed to AppendEntries to {Nonvoter 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca 10.242.129.99:8300}: read tcp 10.242.1.135:58070->10.242.129.99:8300: i/o timeout
2021/05/21 11:44:39 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:44:39 [ERR] raft: Failed to heartbeat to 10.242.129.99:8300: read tcp 10.242.1.135:58074->10.242.129.99:8300: i/o timeout
2021/05/21 11:44:40 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:44:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:48528->10.242.129.99:8302: i/o timeout
2021/05/21 11:44:41 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:44:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:46 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/21 11:44:46 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.129.99
2021/05/21 11:44:46 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/21 11:44:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:48 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:44:48 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.129.99:48928
2021/05/21 11:44:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:49 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:44:49 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/21 11:44:49 [ERR] raft: Failed to AppendEntries to {Nonvoter 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca 10.242.129.99:8300}: read tcp 10.242.1.135:58272->10.242.129.99:8300: i/o timeout
2021/05/21 11:44:49 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:44:50 [ERR] raft: Failed to heartbeat to 10.242.129.99:8300: read tcp 10.242.1.135:58282->10.242.129.99:8300: i/o timeout
2021/05/21 11:44:50 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:44:52 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:44:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:44:59 [ERR] raft: Failed to AppendEntries to {Nonvoter 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca 10.242.129.99:8300}: read tcp 10.242.1.135:58440->10.242.129.99:8300: i/o timeout
2021/05/21 11:44:59 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:45:00 [ERR] raft: Failed to heartbeat to 10.242.129.99:8300: read tcp 10.242.1.135:58480->10.242.129.99:8300: i/o timeout
2021/05/21 11:45:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:01 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:45:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:05 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.129.99:8302
2021/05/21 11:45:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:09 [ERR] raft: Failed to AppendEntries to {Nonvoter 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca 10.242.129.99:8300}: read tcp 10.242.1.135:58628->10.242.129.99:8300: i/o timeout
2021/05/21 11:45:09 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:45:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:11 [ERR] raft: Failed to heartbeat to 10.242.129.99:8300: read tcp 10.242.1.135:58662->10.242.129.99:8300: i/o timeout
2021/05/21 11:45:12 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:45:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:19 [ERR] raft: Failed to AppendEntries to {Nonvoter 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca 10.242.129.99:8300}: read tcp 10.242.1.135:58814->10.242.129.99:8300: i/o timeout
2021/05/21 11:45:20 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:45:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:22 [ERR] raft: Failed to heartbeat to 10.242.129.99:8300: read tcp 10.242.1.135:58858->10.242.129.99:8300: i/o timeout
2021/05/21 11:45:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:23 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:45:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:30 [ERR] raft: Failed to AppendEntries to {Nonvoter 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca 10.242.129.99:8300}: read tcp 10.242.1.135:58992->10.242.129.99:8300: i/o timeout
2021/05/21 11:45:30 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:45:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:31 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:45:31 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/21 11:45:33 [ERR] raft: Failed to heartbeat to 10.242.129.99:8300: read tcp 10.242.1.135:59068->10.242.129.99:8300: i/o timeout
2021/05/21 11:45:34 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:45:34 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/21 11:45:34 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:45:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:40 [ERR] raft: Failed to AppendEntries to {Nonvoter 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca 10.242.129.99:8300}: read tcp 10.242.1.135:59174->10.242.129.99:8300: i/o timeout
2021/05/21 11:45:41 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:45:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:44 [ERR] raft: Failed to heartbeat to 10.242.129.99:8300: read tcp 10.242.1.135:59238->10.242.129.99:8300: i/o timeout
2021/05/21 11:45:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:45 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:45:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:51 [ERR] raft: Failed to AppendEntries to {Nonvoter 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca 10.242.129.99:8300}: read tcp 10.242.1.135:59396->10.242.129.99:8300: i/o timeout
2021/05/21 11:45:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:52 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:45:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:55 [ERR] raft: Failed to heartbeat to 10.242.129.99:8300: read tcp 10.242.1.135:59458->10.242.129.99:8300: i/o timeout
2021/05/21 11:45:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:45:57 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:45:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:02 [ERR] raft: Failed to AppendEntries to {Nonvoter 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca 10.242.129.99:8300}: read tcp 10.242.1.135:59600->10.242.129.99:8300: i/o timeout
2021/05/21 11:46:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:04 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:46:05 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.129.99:8302
2021/05/21 11:46:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:07 [ERR] raft: Failed to heartbeat to 10.242.129.99:8300: read tcp 10.242.1.135:59684->10.242.129.99:8300: i/o timeout
2021/05/21 11:46:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:11 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:46:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:12 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:46:12 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:46:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:13 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:46:13 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/21 11:46:14 [WARN] raft: Failed to contact 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca in 2.500229385s
2021/05/21 11:46:14 [ERR] raft: Failed to AppendEntries to {Nonvoter 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca 10.242.129.99:8300}: read tcp 10.242.1.135:59798->10.242.129.99:8300: i/o timeout
2021/05/21 11:46:16 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:46:17 [WARN] raft: Failed to contact 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca in 4.964247494s
2021/05/21 11:46:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:19 [WARN] raft: Failed to contact 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca in 7.437429336s
2021/05/21 11:46:19 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:46:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:22 [ERR] raft: Failed to heartbeat to 10.242.129.99:8300: read tcp 10.242.1.135:59922->10.242.129.99:8300: i/o timeout
2021/05/21 11:46:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:23 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:46:24 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.99:8301
2021/05/21 11:46:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:29 [ERR] raft: Failed to AppendEntries to {Nonvoter 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca 10.242.129.99:8300}: read tcp 10.242.1.135:60078->10.242.129.99:8300: i/o timeout
2021/05/21 11:46:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:33 [ERR] raft: Failed to heartbeat to 10.242.129.99:8300: read tcp 10.242.1.135:60162->10.242.129.99:8300: i/o timeout
2021/05/21 11:46:34 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:46:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:40 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:46:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:44 [ERR] raft: Failed to heartbeat to 10.242.129.99:8300: read tcp 10.242.1.135:60328->10.242.129.99:8300: i/o timeout
2021/05/21 11:46:44 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:46:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:50 [ERR] raft: Failed to AppendEntries to {Nonvoter 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca 10.242.129.99:8300}: read tcp 10.242.1.135:60450->10.242.129.99:8300: i/o timeout
2021/05/21 11:46:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:54 [ERR] raft: Failed to heartbeat to 10.242.129.99:8300: read tcp 10.242.1.135:60524->10.242.129.99:8300: i/o timeout
2021/05/21 11:46:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:46:55 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:46:55 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/21 11:46:55 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:46:58 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:46:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:00 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:47:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:05 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.129.99:8302
2021/05/21 11:47:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:05 [ERR] raft: Failed to heartbeat to 10.242.129.99:8300: read tcp 10.242.1.135:60736->10.242.129.99:8300: i/o timeout
2021/05/21 11:47:06 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:47:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:07 [DEBUG] raft-net: 10.242.1.135:8300 accepted connection from: 10.242.129.99:33084
2021/05/21 11:47:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:10 [ERR] raft: Failed to AppendEntries to {Nonvoter 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca 10.242.129.99:8300}: read tcp 10.242.1.135:60806->10.242.129.99:8300: i/o timeout
2021/05/21 11:47:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:16 [ERR] raft: Failed to heartbeat to 10.242.129.99:8300: read tcp 10.242.1.135:60932->10.242.129.99:8300: i/o timeout
2021/05/21 11:47:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:17 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:47:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:20 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:47:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:26 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.129.99
2021/05/21 11:47:26 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/21 11:47:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:27 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:47:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:29 [WARN] raft: Failed to contact 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca in 2.500235016s
2021/05/21 11:47:30 [ERR] raft: Failed to AppendEntries to {Nonvoter 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca 10.242.129.99:8300}: read tcp 10.242.1.135:32952->10.242.129.99:8300: i/o timeout
2021/05/21 11:47:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:31 [WARN] raft: Failed to contact 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca in 4.990786514s
2021/05/21 11:47:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:34 [WARN] raft: Failed to contact 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca in 7.486707206s
2021/05/21 11:47:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:37 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:47:37 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/21 11:47:37 [ERR] raft: Failed to heartbeat to 10.242.129.99:8300: read tcp 10.242.1.135:32872->10.242.129.99:8300: i/o timeout
2021/05/21 11:47:38 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:47:39 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:47:39 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.129.99:60680
2021/05/21 11:47:40 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/21 11:47:40 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/21 11:47:40 [WARN] Unable to get address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca, using fallback address 10.242.129.99:8300: Could not find address for server id 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca
2021/05/21 11:47:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:51784->10.242.129.99:8302: i/o timeout
2021/05/21 11:47:41 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:47:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/21 11:47:48 [DEBUG] raft-net: 10.242.1.135:8300 accepted connection from: 10.242.129.99:33406
2021/05/21 11:47:48 [WARN] raft: Rejecting vote request from 10.242.129.99:8300 since we have a leader: 10.242.1.135:8300
2021/05/21 11:47:48 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/21 11:47:48 [ERR] raft: peer {Nonvoter 7e6e5b39-045e-d7a5-b3f5-2ddb9978e6ca 10.242.129.99:8300} has newer term, stopping replication
2021/05/21 11:47:48 [INFO] raft: Node at 10.242.1.135:8300 [Follower] entering Follower state (Leader: "")
2021/05/21 11:47:48 [INFO] raft: aborting pipeline replication to peer {Voter c955779f-89a6-203f-a38f-22883c86e868 10.242.142.201:8300}
2021/05/21 11:47:48 [INFO] consul: cluster leadership lost
2021/05/21 11:47:48 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.99
2021/05/21 11:47:48 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.99:8300) (DC: dc1)
2021/05/21 11:47:48 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/21 11:47:54 [DEBUG] raft-net: 10.242.1.135:8300 accepted connection from: 10.242.142.201:39338
2021/05/21 11:47:54 [INFO] consul: New leader elected: dev-consul-server-2
2021/05/21 11:47:54 [WARN] raft: Rejecting vote request from 10.242.129.99:8300 since we have a leader: 10.242.142.201:8300
2021/05/21 11:47:55 [DEBUG] raft-net: 10.242.1.135:8300 accepted connection from: 10.242.142.201:39354
2021/05/21 11:48:02 [WARN] raft: Rejecting vote request from 10.242.129.99:8300 since we have a leader: 10.242.142.201:8300
2021/05/21 11:48:03 [WARN] raft: Heartbeat timeout from "10.242.142.201:8300" reached, starting election
2021/05/21 11:48:03 [INFO] raft: Node at 10.242.1.135:8300 [Candidate] entering Candidate state in term 8
2021/05/21 11:48:03 [INFO] raft: Node at 10.242.1.135:8300 [Follower] entering Follower state (Leader: "")
2021/05/21 11:48:05 [INFO] consul: New leader elected: dev-consul-server-2
2021/05/21 11:48:06 [DEBUG] raft-net: 10.242.1.135:8300 accepted connection from: 10.242.142.201:39504
2021/05/21 11:48:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:52594->10.242.129.99:8301: i/o timeout
2021/05/21 11:48:17 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:48:20 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/21 11:48:20 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.99
2021/05/21 11:48:20 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.99:8300) (DC: dc1)
2021/05/21 11:48:22 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/21 11:48:22 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.99
2021/05/21 11:48:22 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.99:8300) (DC: dc1)
2021/05/21 11:48:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:52746->10.242.129.99:8301: i/o timeout
2021/05/21 11:48:24 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:48:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:52792->10.242.129.99:8301: i/o timeout
2021/05/21 11:48:27 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:48:28 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/21 11:48:28 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.99
2021/05/21 11:48:28 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.99:8300) (DC: dc1)
2021/05/21 11:48:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:52674->10.242.129.99:8302: i/o timeout
2021/05/21 11:48:34 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.99:8301
2021/05/21 11:48:35 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.99
2021/05/21 11:48:35 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.99:8300) (DC: dc1)
2021/05/21 11:48:36 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/21 11:48:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:53036->10.242.129.99:8301: i/o timeout
2021/05/21 11:48:39 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:48:42 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/21 11:48:42 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.99
2021/05/21 11:48:42 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.99:8300) (DC: dc1)
2021/05/21 11:48:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:53106->10.242.129.99:8301: i/o timeout
2021/05/21 11:48:43 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:48:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:53044->10.242.129.99:8302: i/o timeout
2021/05/21 11:48:51 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:48:56 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.99
2021/05/21 11:48:56 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.99:8300) (DC: dc1)
2021/05/21 11:48:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:53394->10.242.129.99:8301: i/o timeout
2021/05/21 11:48:59 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:49:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:53436->10.242.129.99:8301: i/o timeout
2021/05/21 11:49:02 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:49:04 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.99
2021/05/21 11:49:04 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.99:8300) (DC: dc1)
2021/05/21 11:49:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:53456->10.242.129.99:8301: i/o timeout
2021/05/21 11:49:05 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:49:14 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.99
2021/05/21 11:49:14 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.99:8300) (DC: dc1)
2021/05/21 11:49:17 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/21 11:49:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:53904->10.242.129.99:8301: i/o timeout
2021/05/21 11:49:27 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:49:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:54352->10.242.129.99:8301: i/o timeout
2021/05/21 11:49:51 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:49:53 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/21 11:50:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:54520->10.242.129.99:8301: i/o timeout
2021/05/21 11:50:01 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:50:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:54952->10.242.129.99:8301: i/o timeout
2021/05/21 11:50:24 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:50:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:55008->10.242.129.99:8301: i/o timeout
2021/05/21 11:50:27 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:50:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:55050->10.242.129.99:8301: i/o timeout
2021/05/21 11:50:30 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:50:32 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.99
2021/05/21 11:50:32 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.99:8300) (DC: dc1)
2021/05/21 11:50:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:55148->10.242.129.99:8302: i/o timeout
2021/05/21 11:50:46 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:50:51 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:50:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:55246->10.242.129.99:8302: i/o timeout
2021/05/21 11:50:56 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:51:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:55504->10.242.129.99:8302: i/o timeout
2021/05/21 11:51:06 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:51:10 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/21 11:51:10 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.129.99
2021/05/21 11:51:10 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/21 11:51:23 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.129.99
2021/05/21 11:51:23 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/21 11:51:23 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/21 11:51:24 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.99
2021/05/21 11:51:24 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.99:8300) (DC: dc1)
2021/05/21 11:51:24 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/21 11:51:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56116->10.242.129.99:8301: i/o timeout
2021/05/21 11:51:29 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:51:29 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/21 11:51:29 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.99
2021/05/21 11:51:29 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.99:8300) (DC: dc1)
2021/05/21 11:51:36 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.99:8301
2021/05/21 11:51:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56036->10.242.129.99:8302: i/o timeout
2021/05/21 11:51:36 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:51:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56230->10.242.129.99:8302: i/o timeout
2021/05/21 11:51:46 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:51:51 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.99
2021/05/21 11:51:51 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.99:8300) (DC: dc1)
2021/05/21 11:52:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56674->10.242.129.99:8301: i/o timeout
2021/05/21 11:52:00 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:52:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56506->10.242.129.99:8302: i/o timeout
2021/05/21 11:52:01 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:52:02 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.99
2021/05/21 11:52:02 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.99:8300) (DC: dc1)
2021/05/21 11:52:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56696->10.242.129.99:8301: i/o timeout
2021/05/21 11:52:03 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 11:52:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56592->10.242.129.99:8302: i/o timeout
2021/05/21 11:52:11 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:52:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:56864->10.242.129.99:8302: i/o timeout
2021/05/21 11:52:21 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:52:31 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/21 11:52:31 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.129.99
2021/05/21 11:52:31 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/21 11:52:31 [ERR] memberlist: failed to receive: read tcp 10.242.1.135:8301->10.242.129.99:47740: i/o timeout from=10.242.129.99:47740
2021/05/21 11:52:31 [ERR] memberlist: Failed to send error: write tcp 10.242.1.135:8301->10.242.129.99:47740: i/o timeout from=10.242.129.99:47740
2021/05/21 11:52:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.1.135:57116->10.242.129.99:8302: i/o timeout
2021/05/21 11:52:36 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 11:52:55 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:52:55 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.129.99:36106
2021/05/21 11:52:56 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.129.99:8302
2021/05/21 11:53:16 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.99:8301
2021/05/21 11:53:56 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.129.99:8302
2021/05/21 11:55:56 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.99:8301
2021/05/21 11:55:56 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.129.99:8302
2021/05/21 11:56:56 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.129.99:8302
2021/05/21 11:57:06 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.99:8301
2021/05/21 11:57:26 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 11:58:16 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.99:8301
2021/05/21 11:58:26 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.129.99:8302
2021/05/21 11:58:56 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.99:8301
2021/05/21 11:59:26 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.129.99:8302
2021/05/21 12:00:12 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 12:01:06 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.99:8301
2021/05/21 12:01:56 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.129.99:8302
2021/05/21 12:02:16 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.99:8301
2021/05/21 12:02:56 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.99:8301
2021/05/21 12:02:56 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.129.99:8302
2021/05/21 12:03:48 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 12:04:26 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.129.99:8302
2021/05/21 12:05:26 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.129.99:8302
2021/05/21 12:06:26 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.129.99:8302
2021/05/21 12:06:33 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/21 12:06:36 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.129.99:8301
2021/05/21 12:07:56 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.129.99:8302
2021/05/21 12:08:53 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.129.99
2021/05/21 12:08:53 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/21 12:08:53 [WARN] raft: Rejecting vote request from 10.242.129.99:8300 since we have a leader: 10.242.142.201:8300
2021/05/21 12:08:53 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/21 12:08:54 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.99
2021/05/21 12:08:54 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.99:8300) (DC: dc1)
2021/05/21 12:09:04 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 12:09:05 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/21 12:09:05 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.129.99
2021/05/21 12:09:05 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.129.99:8300) (DC: dc1)
2021/05/21 12:09:07 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/21 12:09:08 [INFO] consul: New leader elected: dev-consul-server-2
2021/05/21 12:09:09 [DEBUG] raft-net: 10.242.1.135:8300 accepted connection from: 10.242.142.201:56398
2021/05/21 12:09:16 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 12:09:40 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/21 12:09:40 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.129.99
2021/05/21 12:09:40 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/21 12:09:54 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/21 12:09:56 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.129.99:8302
2021/05/21 12:09:58 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.129.83
2021/05/21 12:09:58 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.129.83:8300) (DC: dc1)
2021/05/21 12:09:58 [ERR] memberlist: Conflicting address for dev-consul-server-1.dc1. Mine: 10.242.129.99:8302 Theirs: 10.242.129.83:8302
2021/05/21 12:09:58 [WARN] serf: Name conflict for 'dev-consul-server-1.dc1' both 10.242.129.99:8302 and 10.242.129.83:8302 are claiming
2021/05/21 12:09:58 [ERR] memberlist: Conflicting address for dev-consul-server-1.dc1. Mine: 10.242.129.99:8302 Theirs: 10.242.129.83:8302
2021/05/21 12:09:58 [WARN] serf: Name conflict for 'dev-consul-server-1.dc1' both 10.242.129.99:8302 and 10.242.129.83:8302 are claiming
2021/05/21 12:09:58 [ERR] memberlist: Conflicting address for dev-consul-server-1.dc1. Mine: 10.242.129.99:8302 Theirs: 10.242.129.83:8302
2021/05/21 12:09:58 [WARN] serf: Name conflict for 'dev-consul-server-1.dc1' both 10.242.129.99:8302 and 10.242.129.83:8302 are claiming
2021/05/21 12:09:59 [ERR] memberlist: Conflicting address for dev-consul-server-1.dc1. Mine: 10.242.129.99:8302 Theirs: 10.242.129.83:8302
2021/05/21 12:09:59 [WARN] serf: Name conflict for 'dev-consul-server-1.dc1' both 10.242.129.99:8302 and 10.242.129.83:8302 are claiming
2021/05/21 12:09:59 [ERR] memberlist: Conflicting address for dev-consul-server-1.dc1. Mine: 10.242.129.99:8302 Theirs: 10.242.129.83:8302
2021/05/21 12:09:59 [WARN] serf: Name conflict for 'dev-consul-server-1.dc1' both 10.242.129.99:8302 and 10.242.129.83:8302 are claiming
2021/05/21 12:11:13 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.129.83
2021/05/21 12:11:13 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"