By type
+ CONSUL_BIND=
+ [ -n ]
+ CONSUL_CLIENT=
+ [ -n ]
+ CONSUL_DATA_DIR=/consul/data
+ CONSUL_CONFIG_DIR=/consul/config
+ [ -n ]
+ [ a = - ]
+ [ agent = agent ]
+ shift
+ set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
+ [ consul = consul ]
+ stat -c %u /consul/data
+ id -u consul
+ [ 100 != 100 ]
+ stat -c %u /consul/config
+ id -u consul
+ [ 100 != 100 ]
+ [ ! -z ]
+ exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
bootstrap_expect > 0: expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
Version: 'v1.0.6'
Node ID: '07b361e1-37d3-a1ab-60e7-3659ee998414'
Node name: 'dev-consul-server-0'
Datacenter: 'dc1' (Segment: '')
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.242.249.70 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2021/06/13 22:40:06 [INFO] raft: Initial configuration (index=0): []
2021/06/13 22:40:06 [INFO] raft: Node at 10.242.249.70:8300 [Follower] entering Follower state (Leader: "")
2021/06/13 22:40:06 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.249.70
2021/06/13 22:40:06 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.249.70
2021/06/13 22:40:06 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.249.70:8300) (DC: dc1)
2021/06/13 22:40:06 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/06/13 22:40:06 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2021/06/13 22:40:06 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2021/06/13 22:40:06 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
2021/06/13 22:40:06 [INFO] agent: started state syncer
2021/06/13 22:40:06 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer
2021/06/13 22:40:06 [INFO] agent: Joining LAN cluster...
2021/06/13 22:40:06 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/06/13 22:40:06 [WARN] memberlist: Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
2021/06/13 22:40:06 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
2021/06/13 22:40:06 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/06/13 22:40:06 [INFO] agent: (LAN) joined: 0 Err: 3 error(s) occurred:
* Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/06/13 22:40:06 [WARN] agent: Join LAN failed: , retrying in 30s
2021/06/13 22:40:12 [WARN] raft: no known peers, aborting election
2021/06/13 22:40:13 [ERR] agent: failed to sync remote state: No cluster leader
==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=2d46edd1-cd8b-241f-a2f4-3f802fa95b16&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2021/06/13 22:40:19 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.28.14
2021/06/13 22:40:19 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.28.14:8300) (DC: dc1)
2021/06/13 22:40:19 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.28.14
2021/06/13 22:40:19 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/06/13 22:40:29 [ERR] agent: Coordinate update error: No cluster leader
2021/06/13 22:40:35 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-kqtxq 10.242.28.8
2021/06/13 22:40:36 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/06/13 22:40:36 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
2021/06/13 22:40:36 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/06/13 22:40:36 [INFO] agent: (LAN) joined: 1 Err:
2021/06/13 22:40:36 [INFO] agent: Join LAN completed. Synced with 1 initial agents
2021/06/13 22:40:46 [ERR] agent: failed to sync remote state: No cluster leader
2021/06/13 22:40:52 [ERR] agent: Coordinate update error: No cluster leader
2021/06/13 22:41:18 [ERR] agent: Coordinate update error: No cluster leader
2021/06/13 22:41:19 [ERR] agent: failed to sync remote state: No cluster leader
2021/06/13 22:41:40 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.136.142
2021/06/13 22:41:40 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/13 22:41:40 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.242.28.14:8300,10.242.136.142:8300,10.242.249.70:8300
2021/06/13 22:41:40 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.136.142
2021/06/13 22:41:40 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/06/13 22:41:42 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/06/13 22:41:42 [INFO] raft: Node at 10.242.249.70:8300 [Candidate] entering Candidate state in term 2
2021/06/13 22:41:42 [INFO] raft: Election won. Tally: 2
2021/06/13 22:41:42 [INFO] raft: Node at 10.242.249.70:8300 [Leader] entering Leader state
2021/06/13 22:41:42 [INFO] raft: Added peer 23f78c2d-6e31-8c07-eefe-c4fff84e6288, starting replication
2021/06/13 22:41:42 [INFO] raft: Added peer 401b357e-ddab-9e93-04c0-57f882217922, starting replication
2021/06/13 22:41:42 [INFO] consul: cluster leadership acquired
2021/06/13 22:41:42 [INFO] consul: New leader elected: dev-consul-server-0
2021/06/13 22:41:42 [INFO] raft: pipelining replication to peer {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}
2021/06/13 22:41:42 [WARN] raft: AppendEntries to {Voter 23f78c2d-6e31-8c07-eefe-c4fff84e6288 10.242.28.14:8300} rejected, sending older logs (next: 1)
2021/06/13 22:41:42 [INFO] raft: pipelining replication to peer {Voter 23f78c2d-6e31-8c07-eefe-c4fff84e6288 10.242.28.14:8300}
2021/06/13 22:41:42 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/06/13 22:41:42 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/06/13 22:41:42 [INFO] consul: member 'dev-consul-7467bf7b46-kqtxq' joined, marking health alive
2021/06/13 22:41:42 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/06/13 22:41:42 [INFO] agent: Synced node info
2021/06/13 23:04:43 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/13 23:04:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/13 23:05:23 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/13 23:57:45 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/13 23:57:45 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/06/14 00:00:04 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500258621s
2021/06/14 00:00:05 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:00:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:07 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 4.915714856s
2021/06/14 00:00:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:36340->10.242.136.142:8301: i/o timeout
2021/06/14 00:00:07 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:00:08 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/06/14 00:00:08 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.136.142
2021/06/14 00:00:08 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:00:08 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/06/14 00:00:09 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:00:09 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/06/14 00:00:09 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:00:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:10 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500248756s
2021/06/14 00:00:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:13 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 4.955943242s
2021/06/14 00:00:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:15 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 7.434502297s
2021/06/14 00:00:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:51666->10.242.136.142:8302: i/o timeout
2021/06/14 00:00:16 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:00:17 [INFO] raft: aborting pipeline replication to peer {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}
2021/06/14 00:00:17 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:00:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:19 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:35088->10.242.136.142:8300: i/o timeout
2021/06/14 00:00:19 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:00:19 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500195112s
2021/06/14 00:00:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:21 [ERR] yamux: keepalive failed: i/o deadline reached
2021/06/14 00:00:21 [ERR] yamux: keepalive failed: i/o deadline reached
2021/06/14 00:00:21 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/06/14 00:00:21 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.136.142:48224
2021/06/14 00:00:22 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 4.992725898s
2021/06/14 00:00:23 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:00:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:24 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 7.481607234s
2021/06/14 00:00:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:27 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:46956->10.242.136.142:8300: i/o timeout
2021/06/14 00:00:27 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:00:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:29 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:46968->10.242.136.142:8300: i/o timeout
2021/06/14 00:00:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:30 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:00:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:51782->10.242.136.142:8302: i/o timeout
2021/06/14 00:00:31 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:00:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:37 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:47034->10.242.136.142:8300: i/o timeout
2021/06/14 00:00:37 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:00:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:40 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:47054->10.242.136.142:8300: i/o timeout
2021/06/14 00:00:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:51828->10.242.136.142:8302: i/o timeout
2021/06/14 00:00:41 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:00:41 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:00:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:45 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.136.142
2021/06/14 00:00:45 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/06/14 00:00:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:47 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/06/14 00:00:47 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:47124->10.242.136.142:8300: i/o timeout
2021/06/14 00:00:47 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:00:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/06/14 00:00:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:51 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:47152->10.242.136.142:8300: i/o timeout
2021/06/14 00:00:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:52 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:00:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:00:57 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:47212->10.242.136.142:8300: i/o timeout
2021/06/14 00:00:57 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:00:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:02 [ERR] yamux: keepalive failed: i/o deadline reached
2021/06/14 00:01:02 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/06/14 00:01:02 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:47252->10.242.136.142:8300: i/o timeout
2021/06/14 00:01:03 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:01:05 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:01:05 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/06/14 00:01:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:06 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.136.142:8302
2021/06/14 00:01:07 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:47310->10.242.136.142:8300: i/o timeout
2021/06/14 00:01:07 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:01:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:13 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:47352->10.242.136.142:8300: i/o timeout
2021/06/14 00:01:13 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:01:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:17 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:47406->10.242.136.142:8300: i/o timeout
2021/06/14 00:01:17 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:01:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:22 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.136.142
2021/06/14 00:01:22 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/06/14 00:01:23 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:47490->10.242.136.142:8300: i/o timeout
2021/06/14 00:01:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:24 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:01:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:26 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:01:27 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:47544->10.242.136.142:8300: i/o timeout
2021/06/14 00:01:27 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:01:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:28 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:01:29 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:01:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:31 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.50019847s
2021/06/14 00:01:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:33 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 4.919369241s
2021/06/14 00:01:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:35 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/06/14 00:01:35 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.136.142
2021/06/14 00:01:35 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:01:35 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/06/14 00:01:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:52452->10.242.136.142:8302: i/o timeout
2021/06/14 00:01:36 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:01:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:37232->10.242.136.142:8301: i/o timeout
2021/06/14 00:01:37 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:01:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:38 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:47654->10.242.136.142:8300: i/o timeout
2021/06/14 00:01:39 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500165336s
2021/06/14 00:01:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:40 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/06/14 00:01:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:37274->10.242.136.142:8301: i/o timeout
2021/06/14 00:01:41 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:01:41 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/06/14 00:01:41 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.136.142
2021/06/14 00:01:41 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:01:41 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/06/14 00:01:42 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:01:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:43 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500187371s
2021/06/14 00:01:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:46 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.136.142
2021/06/14 00:01:46 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:01:46 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/06/14 00:01:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:48 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:47766->10.242.136.142:8300: i/o timeout
2021/06/14 00:01:49 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/06/14 00:01:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:53 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/06/14 00:01:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:01:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:00 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:47614->10.242.136.142:8300: i/o timeout
2021/06/14 00:02:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:05 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500362039s
2021/06/14 00:02:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:12 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:47898->10.242.136.142:8300: i/o timeout
2021/06/14 00:02:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:19 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:02:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:27 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:02:27 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:48052->10.242.136.142:8300: i/o timeout
2021/06/14 00:02:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:31 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500204162s
2021/06/14 00:02:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:37980->10.242.136.142:8301: i/o timeout
2021/06/14 00:02:33 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:02:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:35 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/06/14 00:02:35 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.136.142
2021/06/14 00:02:35 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:02:35 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/06/14 00:02:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:36 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:02:37 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:02:38 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:02:39 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:02:39 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500185646s
2021/06/14 00:02:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:41 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:02:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:42 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.136.142
2021/06/14 00:02:42 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:02:42 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/06/14 00:02:42 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/06/14 00:02:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:38110->10.242.136.142:8301: i/o timeout
2021/06/14 00:02:43 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:02:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:48 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:48570->10.242.136.142:8300: i/o timeout
2021/06/14 00:02:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:51 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:02:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:38282->10.242.136.142:8301: i/o timeout
2021/06/14 00:02:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:02:56 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500256293s
2021/06/14 00:02:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:02:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:38300->10.242.136.142:8301: i/o timeout
2021/06/14 00:02:58 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:02:58 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500249088s
2021/06/14 00:02:59 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/06/14 00:02:59 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.136.142
2021/06/14 00:02:59 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:02:59 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/06/14 00:02:59 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:03:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:00 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:03:01 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:03:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:02 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:03:03 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:03:04 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:03:05 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:03:05 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/06/14 00:03:05 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.136.142
2021/06/14 00:03:05 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:03:05 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/06/14 00:03:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:08 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:48852->10.242.136.142:8300: i/o timeout
2021/06/14 00:03:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:38542->10.242.136.142:8301: i/o timeout
2021/06/14 00:03:11 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:03:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:14 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500335046s
2021/06/14 00:03:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:38606->10.242.136.142:8301: i/o timeout
2021/06/14 00:03:16 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:03:17 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 4.992368317s
2021/06/14 00:03:17 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/06/14 00:03:17 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.136.142
2021/06/14 00:03:17 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:03:17 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/06/14 00:03:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:18 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:03:19 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 7.412864597s
2021/06/14 00:03:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:53904->10.242.136.142:8302: i/o timeout
2021/06/14 00:03:21 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:03:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:23 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:48290->10.242.136.142:8300: i/o timeout
2021/06/14 00:03:23 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:03:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:28 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:49158->10.242.136.142:8300: i/o timeout
2021/06/14 00:03:29 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:03:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:31 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/06/14 00:03:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:33 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:49242->10.242.136.142:8300: i/o timeout
2021/06/14 00:03:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:34 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:03:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:54134->10.242.136.142:8302: i/o timeout
2021/06/14 00:03:36 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:03:36 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.136.142:8301
2021/06/14 00:03:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:38 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/06/14 00:03:38 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:03:38 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:03:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:40 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500235346s
2021/06/14 00:03:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:42 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 4.981648116s
2021/06/14 00:03:43 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.136.142
2021/06/14 00:03:43 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:03:43 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/06/14 00:03:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:39040->10.242.136.142:8301: i/o timeout
2021/06/14 00:03:45 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:03:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:46 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500173558s
2021/06/14 00:03:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:39070->10.242.136.142:8301: i/o timeout
2021/06/14 00:03:48 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:03:48 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:49466->10.242.136.142:8300: i/o timeout
2021/06/14 00:03:48 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 4.994421223s
2021/06/14 00:03:49 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.136.142
2021/06/14 00:03:49 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:03:49 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/06/14 00:03:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:54356->10.242.136.142:8302: i/o timeout
2021/06/14 00:03:51 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:03:51 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 7.493992068s
2021/06/14 00:03:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:53 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:03:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:03:55 [DEBUG] raft-net: 10.242.249.70:8300 accepted connection from: 10.242.136.142:45604
2021/06/14 00:03:55 [WARN] raft: Rejecting vote request from 10.242.136.142:8300 since we have a leader: 10.242.249.70:8300
2021/06/14 00:03:55 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.136.142
2021/06/14 00:03:55 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:03:55 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/06/14 00:03:59 [INFO] raft: pipelining replication to peer {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}
2021/06/14 00:04:03 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:04:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:39320->10.242.136.142:8301: i/o timeout
2021/06/14 00:04:03 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:04:03 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500328834s
2021/06/14 00:04:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:08:15 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:08:19 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:08:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:08:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:43384->10.242.136.142:8301: i/o timeout
2021/06/14 00:08:29 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:08:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:08:30 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500285267s
2021/06/14 00:08:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:08:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:43494->10.242.136.142:8301: i/o timeout
2021/06/14 00:08:33 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:08:33 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 4.987514007s
2021/06/14 00:08:33 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/06/14 00:08:33 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.136.142
2021/06/14 00:08:33 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:08:33 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/06/14 00:08:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:08:35 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 7.41434389s
2021/06/14 00:08:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:08:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:58772->10.242.136.142:8302: i/o timeout
2021/06/14 00:08:36 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:08:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:43510->10.242.136.142:8301: i/o timeout
2021/06/14 00:08:36 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:08:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:08:38 [INFO] raft: aborting pipeline replication to peer {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}
2021/06/14 00:08:38 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:08:38 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.136.142:8301
2021/06/14 00:08:38 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:49774->10.242.136.142:8300: i/o timeout
2021/06/14 00:08:39 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.136.142
2021/06/14 00:08:39 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:08:39 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/06/14 00:08:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:08:40 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500188601s
2021/06/14 00:08:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:43630->10.242.136.142:8301: i/o timeout
2021/06/14 00:08:41 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:08:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:08:43 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 4.98137163s
2021/06/14 00:08:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:08:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:43646->10.242.136.142:8301: i/o timeout
2021/06/14 00:08:44 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:08:45 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/06/14 00:08:45 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.136.142
2021/06/14 00:08:45 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:08:45 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/06/14 00:08:45 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 7.417254783s
2021/06/14 00:08:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:08:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:58932->10.242.136.142:8302: i/o timeout
2021/06/14 00:08:46 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:08:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:08:48 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:54100->10.242.136.142:8300: i/o timeout
2021/06/14 00:08:48 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:08:48 [WARN] raft: Rejecting vote request from 10.242.136.142:8300 since we have a leader: 10.242.249.70:8300
2021/06/14 00:08:49 [ERR] raft: peer {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300} has newer term, stopping replication
2021/06/14 00:08:49 [INFO] raft: Node at 10.242.249.70:8300 [Follower] entering Follower state (Leader: "")
2021/06/14 00:08:49 [INFO] raft: aborting pipeline replication to peer {Voter 23f78c2d-6e31-8c07-eefe-c4fff84e6288 10.242.28.14:8300}
2021/06/14 00:08:49 [INFO] consul: cluster leadership lost
2021/06/14 00:08:49 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.136.142
2021/06/14 00:08:49 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:08:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:43802->10.242.136.142:8301: i/o timeout
2021/06/14 00:08:52 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:08:54 [WARN] consul.coordinate: Batch update failed: node is not the leader
2021/06/14 00:08:55 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/06/14 00:08:55 [INFO] raft: Node at 10.242.249.70:8300 [Candidate] entering Candidate state in term 3
2021/06/14 00:08:55 [INFO] raft: Election won. Tally: 2
2021/06/14 00:08:55 [INFO] raft: Node at 10.242.249.70:8300 [Leader] entering Leader state
2021/06/14 00:08:55 [INFO] raft: Added peer 23f78c2d-6e31-8c07-eefe-c4fff84e6288, starting replication
2021/06/14 00:08:55 [INFO] raft: Added peer 401b357e-ddab-9e93-04c0-57f882217922, starting replication
2021/06/14 00:08:55 [INFO] consul: cluster leadership acquired
2021/06/14 00:08:55 [INFO] consul: New leader elected: dev-consul-server-0
2021/06/14 00:08:55 [INFO] raft: pipelining replication to peer {Voter 23f78c2d-6e31-8c07-eefe-c4fff84e6288 10.242.28.14:8300}
2021/06/14 00:08:55 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/06/14 00:08:56 [WARN] raft: AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300} rejected, sending older logs (next: 1003)
2021/06/14 00:08:57 [INFO] raft: pipelining replication to peer {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}
2021/06/14 00:08:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:08:58 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/06/14 00:08:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:04 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:09:06 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:09:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:44050->10.242.136.142:8301: i/o timeout
2021/06/14 00:09:08 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:09:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:09:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:09 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500244937s
2021/06/14 00:09:10 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.136.142
2021/06/14 00:09:10 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:09:10 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/06/14 00:09:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:44076->10.242.136.142:8301: i/o timeout
2021/06/14 00:09:11 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:09:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:12 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 4.974160246s
2021/06/14 00:09:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:14 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 7.473759116s
2021/06/14 00:09:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:59414->10.242.136.142:8302: i/o timeout
2021/06/14 00:09:16 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:09:17 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:09:17 [INFO] raft: aborting pipeline replication to peer {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}
2021/06/14 00:09:17 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:09:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:19 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500186952s
2021/06/14 00:09:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:22 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 4.981599995s
2021/06/14 00:09:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:24 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 7.411762349s
2021/06/14 00:09:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:59492->10.242.136.142:8302: i/o timeout
2021/06/14 00:09:26 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:09:27 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:54388->10.242.136.142:8300: i/o timeout
2021/06/14 00:09:27 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:54260->10.242.136.142:8300: i/o timeout
2021/06/14 00:09:27 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:09:27 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:09:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:34 [ERR] memberlist: failed to receive: read tcp 10.242.249.70:8301->10.242.136.142:60268: i/o timeout from=10.242.136.142:60268
2021/06/14 00:09:34 [ERR] memberlist: Failed to send error: write tcp 10.242.249.70:8301->10.242.136.142:60268: i/o timeout from=10.242.136.142:60268
2021/06/14 00:09:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:36 [ERR] yamux: keepalive failed: i/o deadline reached
2021/06/14 00:09:36 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.136.142:43626
2021/06/14 00:09:37 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:54886->10.242.136.142:8300: i/o timeout
2021/06/14 00:09:37 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:09:37 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:54892->10.242.136.142:8300: i/o timeout
2021/06/14 00:09:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:38 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:09:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:59806->10.242.136.142:8302: i/o timeout
2021/06/14 00:09:41 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:09:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:45 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.136.142
2021/06/14 00:09:45 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/06/14 00:09:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:47 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:55040->10.242.136.142:8300: i/o timeout
2021/06/14 00:09:47 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:09:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:48 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:55058->10.242.136.142:8300: i/o timeout
2021/06/14 00:09:49 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:09:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:09:56 [ERR] yamux: keepalive failed: i/o deadline reached
2021/06/14 00:09:56 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/06/14 00:09:57 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:55194->10.242.136.142:8300: i/o timeout
2021/06/14 00:09:57 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:09:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:09:59 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:55224->10.242.136.142:8300: i/o timeout
2021/06/14 00:09:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:00 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:10:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:07 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:55352->10.242.136.142:8300: i/o timeout
2021/06/14 00:10:07 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:10:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:10 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:55390->10.242.136.142:8300: i/o timeout
2021/06/14 00:10:10 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:10:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:17 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:55510->10.242.136.142:8300: i/o timeout
2021/06/14 00:10:17 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:10:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:20 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:55566->10.242.136.142:8300: i/o timeout
2021/06/14 00:10:21 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:10:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:27 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:55688->10.242.136.142:8300: i/o timeout
2021/06/14 00:10:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:28 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:10:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:31 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:55730->10.242.136.142:8300: i/o timeout
2021/06/14 00:10:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:32 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:10:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:35 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.136.142:8302
2021/06/14 00:10:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:37 [ERR] yamux: keepalive failed: i/o deadline reached
2021/06/14 00:10:37 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/06/14 00:10:38 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:55836->10.242.136.142:8300: i/o timeout
2021/06/14 00:10:38 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:10:40 [WARN] raft: Rejecting vote request from 10.242.136.142:8300 since we have a leader: 10.242.249.70:8300
2021/06/14 00:10:40 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:10:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:42 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:55908->10.242.136.142:8300: i/o timeout
2021/06/14 00:10:43 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:10:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:48 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:56002->10.242.136.142:8300: i/o timeout
2021/06/14 00:10:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:10:49 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:10:51 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.136.142
2021/06/14 00:10:51 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/06/14 00:10:51 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.136.142
2021/06/14 00:10:51 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:10:51 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/06/14 00:10:51 [ERR] raft: peer {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300} has newer term, stopping replication
2021/06/14 00:10:51 [INFO] raft: Node at 10.242.249.70:8300 [Follower] entering Follower state (Leader: "")
2021/06/14 00:10:51 [INFO] raft: aborting pipeline replication to peer {Voter 23f78c2d-6e31-8c07-eefe-c4fff84e6288 10.242.28.14:8300}
2021/06/14 00:10:51 [INFO] consul: cluster leadership lost
2021/06/14 00:10:51 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/06/14 00:10:51 [WARN] raft: Rejecting vote request from 10.242.136.142:8300 since our last index is greater (1030, 1013)
2021/06/14 00:10:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:45710->10.242.136.142:8301: i/o timeout
2021/06/14 00:10:54 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:10:56 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/06/14 00:10:56 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.136.142
2021/06/14 00:10:56 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:10:57 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/06/14 00:10:57 [INFO] raft: Node at 10.242.249.70:8300 [Candidate] entering Candidate state in term 6
2021/06/14 00:10:57 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:10:57 [INFO] raft: Election won. Tally: 2
2021/06/14 00:10:57 [INFO] raft: Node at 10.242.249.70:8300 [Leader] entering Leader state
2021/06/14 00:10:57 [INFO] raft: Added peer 23f78c2d-6e31-8c07-eefe-c4fff84e6288, starting replication
2021/06/14 00:10:57 [INFO] consul: cluster leadership acquired
2021/06/14 00:10:57 [INFO] consul: New leader elected: dev-consul-server-0
2021/06/14 00:10:57 [INFO] raft: Added peer 401b357e-ddab-9e93-04c0-57f882217922, starting replication
2021/06/14 00:10:57 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:10:57 [INFO] raft: pipelining replication to peer {Voter 23f78c2d-6e31-8c07-eefe-c4fff84e6288 10.242.28.14:8300}
2021/06/14 00:10:57 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/06/14 00:10:58 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:10:58 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.136.142
2021/06/14 00:10:58 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:10:58 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/06/14 00:11:00 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.501988138s
2021/06/14 00:11:00 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:11:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:45852->10.242.136.142:8301: i/o timeout
2021/06/14 00:11:02 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:11:02 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 4.957351098s
2021/06/14 00:11:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:03 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/06/14 00:11:03 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.136.142
2021/06/14 00:11:03 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:11:03 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/06/14 00:11:04 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 7.434189878s
2021/06/14 00:11:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:07 [ERR] raft: Failed to make RequestVote RPC to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:56172->10.242.136.142:8300: i/o timeout
2021/06/14 00:11:07 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:56080->10.242.136.142:8300: i/o timeout
2021/06/14 00:11:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:07 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:11:08 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:56318->10.242.136.142:8300: i/o timeout
2021/06/14 00:11:09 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:11:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:32992->10.242.136.142:8302: i/o timeout
2021/06/14 00:11:11 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:11:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:17 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:56464->10.242.136.142:8300: i/o timeout
2021/06/14 00:11:17 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:11:19 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:56482->10.242.136.142:8300: i/o timeout
2021/06/14 00:11:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:19 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:11:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:33240->10.242.136.142:8302: i/o timeout
2021/06/14 00:11:26 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:11:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:27 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:56636->10.242.136.142:8300: i/o timeout
2021/06/14 00:11:27 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:11:28 [ERR] yamux: keepalive failed: i/o deadline reached
2021/06/14 00:11:28 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.136.142:52506
2021/06/14 00:11:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:29 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:56660->10.242.136.142:8300: i/o timeout
2021/06/14 00:11:30 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:11:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:33394->10.242.136.142:8302: i/o timeout
2021/06/14 00:11:36 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:11:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:37 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:56788->10.242.136.142:8300: i/o timeout
2021/06/14 00:11:37 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:11:39 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.136.142:8301
2021/06/14 00:11:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:40 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:56832->10.242.136.142:8300: i/o timeout
2021/06/14 00:11:41 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:11:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:33554->10.242.136.142:8302: i/o timeout
2021/06/14 00:11:46 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:11:46 [ERR] memberlist: Push/Pull with dev-consul-server-2.dc1 failed: read tcp 10.242.249.70:33110->10.242.136.142:8302: i/o timeout
2021/06/14 00:11:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:47 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:56942->10.242.136.142:8300: i/o timeout
2021/06/14 00:11:47 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:11:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:51 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:57002->10.242.136.142:8300: i/o timeout
2021/06/14 00:11:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:52 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:11:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:54 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.136.142
2021/06/14 00:11:54 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/06/14 00:11:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:33634->10.242.136.142:8302: i/o timeout
2021/06/14 00:11:56 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:11:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:11:57 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:57108->10.242.136.142:8300: i/o timeout
2021/06/14 00:11:57 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:11:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:02 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:57178->10.242.136.142:8300: i/o timeout
2021/06/14 00:12:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:03 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:12:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:07 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:57264->10.242.136.142:8300: i/o timeout
2021/06/14 00:12:08 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:12:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:13 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:57354->10.242.136.142:8300: i/o timeout
2021/06/14 00:12:14 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:12:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:18 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:57424->10.242.136.142:8300: i/o timeout
2021/06/14 00:12:18 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:12:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:19 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:12:21 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.136.142:8302
2021/06/14 00:12:21 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.50032445s
2021/06/14 00:12:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:23 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 4.978761248s
2021/06/14 00:12:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:25 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.136.142
2021/06/14 00:12:25 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:12:25 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/06/14 00:12:26 [WARN] raft: AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300} rejected, sending older logs (next: 1014)
2021/06/14 00:12:28 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:12:28 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/06/14 00:12:28 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/06/14 00:12:28 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500258995s
2021/06/14 00:12:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:29 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:57536->10.242.136.142:8300: i/o timeout
2021/06/14 00:12:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:47250->10.242.136.142:8301: i/o timeout
2021/06/14 00:12:30 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:12:31 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 4.922648684s
2021/06/14 00:12:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:31 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/06/14 00:12:31 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.136.142
2021/06/14 00:12:31 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:12:31 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/06/14 00:12:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:47272->10.242.136.142:8301: i/o timeout
2021/06/14 00:12:33 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:12:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:33 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 7.386417054s
2021/06/14 00:12:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:36 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:57602->10.242.136.142:8300: i/o timeout
2021/06/14 00:12:36 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:12:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:40 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:57784->10.242.136.142:8300: i/o timeout
2021/06/14 00:12:41 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:12:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:46 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:57870->10.242.136.142:8300: i/o timeout
2021/06/14 00:12:46 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:12:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:51 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:57954->10.242.136.142:8300: i/o timeout
2021/06/14 00:12:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:52 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:12:53 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.136.142
2021/06/14 00:12:53 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/06/14 00:12:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:12:56 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:58022->10.242.136.142:8300: i/o timeout
2021/06/14 00:12:56 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:12:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:12:58 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:12:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:13:00 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500180133s
2021/06/14 00:13:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:34748->10.242.136.142:8302: i/o timeout
2021/06/14 00:13:01 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:13:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:13:02 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.136.142
2021/06/14 00:13:02 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:13:02 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/06/14 00:13:02 [INFO] raft: pipelining replication to peer {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}
2021/06/14 00:14:42 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:14:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:49412->10.242.136.142:8301: i/o timeout
2021/06/14 00:14:48 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:14:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:14:54 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:14:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:15:46 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:16:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:50816->10.242.136.142:8301: i/o timeout
2021/06/14 00:16:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:16:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:16:16 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:16:16 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/06/14 00:16:20 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:16:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:16:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:16:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:16:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:51062->10.242.136.142:8301: i/o timeout
2021/06/14 00:16:27 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:16:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:51144->10.242.136.142:8301: i/o timeout
2021/06/14 00:16:31 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:16:32 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:16:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:51302->10.242.136.142:8301: i/o timeout
2021/06/14 00:16:41 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:16:48 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:16:48 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/06/14 00:16:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:51420->10.242.136.142:8301: i/o timeout
2021/06/14 00:16:49 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:16:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:16:49 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500254379s
2021/06/14 00:16:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:16:52 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 4.958491316s
2021/06/14 00:16:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:51486->10.242.136.142:8301: i/o timeout
2021/06/14 00:16:52 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:16:53 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.136.142
2021/06/14 00:16:53 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:16:53 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/06/14 00:16:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:16:54 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 7.43459213s
2021/06/14 00:16:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:16:55 [INFO] raft: aborting pipeline replication to peer {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}
2021/06/14 00:16:56 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:16:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:38542->10.242.136.142:8302: i/o timeout
2021/06/14 00:16:56 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:16:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:16:57 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:58128->10.242.136.142:8300: i/o timeout
2021/06/14 00:16:58 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:16:58 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500180602s
2021/06/14 00:16:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:17:00 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:17:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:17:02 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:17:02 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/06/14 00:17:02 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:17:02 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.136.142
2021/06/14 00:17:02 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:17:02 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/06/14 00:17:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:17:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:17:06 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:33830->10.242.136.142:8300: i/o timeout
2021/06/14 00:17:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:17:08 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500223777s
2021/06/14 00:17:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:17:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:17:13 [INFO] raft: pipelining replication to peer {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}
2021/06/14 00:17:42 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:17:42 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/06/14 00:17:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:17:44 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/06/14 00:17:45 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/06/14 00:17:47 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/06/14 00:19:18 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:19:18 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/06/14 00:19:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:19:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:19:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:54036->10.242.136.142:8301: i/o timeout
2021/06/14 00:19:24 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:19:28 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:19:28 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500209961s
2021/06/14 00:19:48 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:19:52 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:19:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:19:56 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:19:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:19:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:54630->10.242.136.142:8301: i/o timeout
2021/06/14 00:19:58 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:19:58 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.50027018s
2021/06/14 00:19:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:20:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:54654->10.242.136.142:8301: i/o timeout
2021/06/14 00:20:01 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:20:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:20:01 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/06/14 00:20:01 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.136.142
2021/06/14 00:20:01 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:20:01 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/06/14 00:20:02 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:20:02 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:20:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:20:04 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.136.142
2021/06/14 00:20:04 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:20:04 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/06/14 00:20:06 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:20:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:55464->10.242.136.142:8301: i/o timeout
2021/06/14 00:20:49 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:20:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:20:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:20:56 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:20:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:55646->10.242.136.142:8301: i/o timeout
2021/06/14 00:21:00 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:21:00 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:21:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:06 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:21:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:55768->10.242.136.142:8301: i/o timeout
2021/06/14 00:21:07 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:21:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:08 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500232114s
2021/06/14 00:21:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:55788->10.242.136.142:8301: i/o timeout
2021/06/14 00:21:10 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:21:11 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 4.914750024s
2021/06/14 00:21:11 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/06/14 00:21:11 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.136.142
2021/06/14 00:21:11 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:21:11 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/06/14 00:21:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:13 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 7.339575342s
2021/06/14 00:21:15 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:21:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:16 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:21:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:42922->10.242.136.142:8302: i/o timeout
2021/06/14 00:21:16 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:21:16 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:21:18 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:21:18 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:21:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:20 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:21:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:22 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500234444s
2021/06/14 00:21:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:24 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 4.983646422s
2021/06/14 00:21:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:26 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:21:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:27 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500328421s
2021/06/14 00:21:28 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.136.142
2021/06/14 00:21:28 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:21:28 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/06/14 00:21:28 [ERR] raft: Failed to pipeline AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: write tcp 10.242.249.70:33872->10.242.136.142:8300: use of closed network connection
2021/06/14 00:21:28 [INFO] raft: aborting pipeline replication to peer {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}
2021/06/14 00:21:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:30 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/06/14 00:21:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:33 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500265985s
2021/06/14 00:21:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:56228->10.242.136.142:8301: i/o timeout
2021/06/14 00:21:34 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:21:36 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:21:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:56298->10.242.136.142:8301: i/o timeout
2021/06/14 00:21:38 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:21:38 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/06/14 00:21:38 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.136.142
2021/06/14 00:21:38 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:21:38 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/06/14 00:21:38 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:38422->10.242.136.142:8300: i/o timeout
2021/06/14 00:21:38 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:21:39 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500256527s
2021/06/14 00:21:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:43340->10.242.136.142:8302: i/o timeout
2021/06/14 00:21:41 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:21:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:41 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 4.984503593s
2021/06/14 00:21:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:44 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 7.393231676s
2021/06/14 00:21:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:47 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:33992->10.242.136.142:8300: i/o timeout
2021/06/14 00:21:48 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:21:48 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:38588->10.242.136.142:8300: i/o timeout
2021/06/14 00:21:48 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:21:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:43594->10.242.136.142:8302: i/o timeout
2021/06/14 00:21:56 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:21:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:21:58 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:38756->10.242.136.142:8300: i/o timeout
2021/06/14 00:21:58 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:38764->10.242.136.142:8300: i/o timeout
2021/06/14 00:21:58 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:21:59 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:21:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:22:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:22:01 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:22:03 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500290704s
2021/06/14 00:22:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:22:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:22:05 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 4.945212518s
2021/06/14 00:22:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:43678->10.242.136.142:8302: i/o timeout
2021/06/14 00:22:06 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:22:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:22:08 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 7.400991917s
2021/06/14 00:22:08 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:38928->10.242.136.142:8300: i/o timeout
2021/06/14 00:22:08 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:22:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:22:09 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/06/14 00:22:09 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.136.142
2021/06/14 00:22:09 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/06/14 00:22:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:22:11 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:38938->10.242.136.142:8300: i/o timeout
2021/06/14 00:22:12 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:22:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:22:14 [ERR] yamux: keepalive failed: i/o deadline reached
2021/06/14 00:22:14 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/06/14 00:22:14 [ERR] yamux: keepalive failed: i/o deadline reached
2021/06/14 00:22:14 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.136.142:53772
2021/06/14 00:22:16 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:22:16 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/06/14 00:22:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:22:18 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:39098->10.242.136.142:8300: i/o timeout
2021/06/14 00:22:18 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:22:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:22:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:22:22 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:39174->10.242.136.142:8300: i/o timeout
2021/06/14 00:22:23 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:22:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:22:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:22:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:22:28 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:39280->10.242.136.142:8300: i/o timeout
2021/06/14 00:22:29 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:22:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:22:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:22:33 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:39350->10.242.136.142:8300: i/o timeout
2021/06/14 00:22:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:22:34 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:22:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:22:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:22:39 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:39446->10.242.136.142:8300: i/o timeout
2021/06/14 00:22:39 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:22:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:22:41 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.136.142
2021/06/14 00:22:41 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:22:41 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/06/14 00:22:41 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.136.142
2021/06/14 00:22:41 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/06/14 00:22:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:22:43 [INFO] raft: pipelining replication to peer {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}
2021/06/14 00:22:46 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:22:46 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/06/14 00:22:56 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:24:56 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:26:02 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:26:06 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:26:18 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:26:22 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:26:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:26:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:26:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:32928->10.242.136.142:8301: i/o timeout
2021/06/14 00:26:28 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:26:29 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500288605s
2021/06/14 00:26:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:26:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:32976->10.242.136.142:8301: i/o timeout
2021/06/14 00:26:31 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:26:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:26:32 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/06/14 00:26:32 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.136.142
2021/06/14 00:26:32 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:26:32 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/06/14 00:26:32 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:26:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:26:33 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:26:34 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:26:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:26:36 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500256651s
2021/06/14 00:26:36 [INFO] raft: aborting pipeline replication to peer {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}
2021/06/14 00:26:36 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:26:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:26:38 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:26:38 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.136.142
2021/06/14 00:26:38 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:26:38 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/06/14 00:26:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:26:41 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500313213s
2021/06/14 00:26:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:26:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:33160->10.242.136.142:8301: i/o timeout
2021/06/14 00:26:42 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:26:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:26:43 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 4.991852494s
2021/06/14 00:26:44 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/06/14 00:26:44 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.136.142
2021/06/14 00:26:44 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:26:44 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/06/14 00:26:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:26:46 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 7.432780145s
2021/06/14 00:26:46 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:43588->10.242.136.142:8300: i/o timeout
2021/06/14 00:26:46 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:26:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:26:49 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:39538->10.242.136.142:8300: i/o timeout
2021/06/14 00:26:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:26:50 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:26:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:48538->10.242.136.142:8302: i/o timeout
2021/06/14 00:26:51 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:26:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:26:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:26:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:26:56 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:43752->10.242.136.142:8300: i/o timeout
2021/06/14 00:26:56 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:26:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:26:58 [ERR] yamux: keepalive failed: i/o deadline reached
2021/06/14 00:26:58 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/06/14 00:27:00 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:43812->10.242.136.142:8300: i/o timeout
2021/06/14 00:27:00 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:27:00 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:27:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:48708->10.242.136.142:8302: i/o timeout
2021/06/14 00:27:01 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:27:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:27:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:27:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:27:06 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:43932->10.242.136.142:8300: i/o timeout
2021/06/14 00:27:06 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:27:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:27:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:27:10 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:44004->10.242.136.142:8300: i/o timeout
2021/06/14 00:27:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:48800->10.242.136.142:8302: i/o timeout
2021/06/14 00:27:11 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:27:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:27:11 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:27:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:27:14 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/06/14 00:27:14 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.136.142
2021/06/14 00:27:14 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/06/14 00:27:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:27:16 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:44104->10.242.136.142:8300: i/o timeout
2021/06/14 00:27:16 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:27:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:27:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:27:21 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.136.142:8302
2021/06/14 00:27:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:27:21 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:44208->10.242.136.142:8300: i/o timeout
2021/06/14 00:27:22 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:27:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:27:24 [ERR] yamux: keepalive failed: i/o deadline reached
2021/06/14 00:27:24 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.136.142:35436
2021/06/14 00:27:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:27:26 [ERR] raft: Failed to AppendEntries to {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}: read tcp 10.242.249.70:44294->10.242.136.142:8300: i/o timeout
2021/06/14 00:27:26 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:27:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:27:27 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:27:28 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/06/14 00:27:28 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.136.142
2021/06/14 00:27:28 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:27:28 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/06/14 00:27:28 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.136.142
2021/06/14 00:27:28 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/06/14 00:27:28 [INFO] raft: pipelining replication to peer {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}
2021/06/14 00:27:31 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/06/14 00:33:46 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:33:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:33:50 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/06/14 00:33:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:33:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:40432->10.242.136.142:8301: i/o timeout
2021/06/14 00:33:54 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:33:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:33:58 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/06/14 00:33:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:33:59 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500165719s
2021/06/14 00:34:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:40578->10.242.136.142:8301: i/o timeout
2021/06/14 00:34:01 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:34:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:34:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:34:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:40624->10.242.136.142:8301: i/o timeout
2021/06/14 00:34:04 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:34:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:34:07 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/06/14 00:34:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:34:09 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.136.142
2021/06/14 00:34:09 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:34:09 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/06/14 00:34:09 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:34:10 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.136.142
2021/06/14 00:34:10 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:34:10 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/06/14 00:34:10 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:34:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:40780->10.242.136.142:8301: i/o timeout
2021/06/14 00:34:12 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:34:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:34:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:34:15 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500215312s
2021/06/14 00:34:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:40828->10.242.136.142:8301: i/o timeout
2021/06/14 00:34:15 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:34:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:34:16 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.136.142
2021/06/14 00:34:16 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:34:16 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/06/14 00:34:16 [WARN] Unable to get address for server id 401b357e-ddab-9e93-04c0-57f882217922, using fallback address 10.242.136.142:8300: Could not find address for server id 401b357e-ddab-9e93-04c0-57f882217922
2021/06/14 00:34:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:34:18 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500233387s
2021/06/14 00:34:19 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.136.142
2021/06/14 00:34:19 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:34:19 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/06/14 00:34:20 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/06/14 00:34:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/06/14 00:34:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:34:21 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.500280017s
2021/06/14 00:34:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:40946->10.242.136.142:8301: i/o timeout
2021/06/14 00:34:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:34:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:34:24 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 4.999136593s
2021/06/14 00:34:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:40998->10.242.136.142:8301: i/o timeout
2021/06/14 00:34:25 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:34:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:34:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:56238->10.242.136.142:8302: i/o timeout
2021/06/14 00:34:26 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:34:26 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 7.475282636s
2021/06/14 00:34:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:34:29 [INFO] raft: aborting pipeline replication to peer {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}
2021/06/14 00:34:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:41062->10.242.136.142:8301: i/o timeout
2021/06/14 00:34:29 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:34:29 [ERR] raft: Failed to heartbeat to 10.242.136.142:8300: read tcp 10.242.249.70:44392->10.242.136.142:8300: i/o timeout
2021/06/14 00:34:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:34:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:34:31 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 2.50018799s
2021/06/14 00:34:31 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/06/14 00:34:31 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.136.142
2021/06/14 00:34:31 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:34:31 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/06/14 00:34:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:41082->10.242.136.142:8301: i/o timeout
2021/06/14 00:34:32 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/06/14 00:34:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:34:34 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 4.973481165s
2021/06/14 00:34:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/06/14 00:34:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.249.70:56416->10.242.136.142:8302: i/o timeout
2021/06/14 00:34:36 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/06/14 00:34:36 [WARN] raft: Failed to contact 401b357e-ddab-9e93-04c0-57f882217922 in 7.44693222s
2021/06/14 00:34:37 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.136.142
2021/06/14 00:34:37 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.136.142:8300) (DC: dc1)
2021/06/14 00:34:37 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/06/14 00:34:37 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/06/14 00:34:37 [INFO] raft: pipelining replication to peer {Voter 401b357e-ddab-9e93-04c0-57f882217922 10.242.136.142:8300}