By type
+ CONSUL_BIND=
+ [ -n ]
+ CONSUL_CLIENT=
+ [ -n ]
+ CONSUL_DATA_DIR=/consul/data
+ CONSUL_CONFIG_DIR=/consul/config
+ [ -n ]
+ [ a = - ]
+ [ agent = agent ]
+ shift
+ set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
+ [ consul = consul ]
+ stat -c %u /consul/data
+ id -u consul
+ [ 100 != 100 ]
+ stat -c %u /consul/config
+ id -u consul
+ [ 100 != 100 ]
+ [ ! -z ]
+ exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
bootstrap_expect > 0: expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
Version: 'v1.0.6'
Node ID: '6ff779d7-507c-f4ae-1d24-7621a2e1d40c'
Node name: 'dev-consul-server-0'
Datacenter: 'dc1' (Segment: '')
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.242.61.70 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2021/05/09 22:39:28 [INFO] raft: Initial configuration (index=0): []
2021/05/09 22:39:28 [INFO] raft: Node at 10.242.61.70:8300 [Follower] entering Follower state (Leader: "")
2021/05/09 22:39:28 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.61.70
2021/05/09 22:39:28 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.61.70
2021/05/09 22:39:28 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/09 22:39:28 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.61.70:8300) (DC: dc1)
2021/05/09 22:39:28 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2021/05/09 22:39:28 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2021/05/09 22:39:28 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
2021/05/09 22:39:28 [INFO] agent: started state syncer
2021/05/09 22:39:28 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer
2021/05/09 22:39:28 [INFO] agent: Joining LAN cluster...
2021/05/09 22:39:28 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/05/09 22:39:28 [WARN] memberlist: Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
2021/05/09 22:39:28 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
2021/05/09 22:39:29 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/05/09 22:39:29 [INFO] agent: (LAN) joined: 0 Err: 3 error(s) occurred:
* Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/05/09 22:39:29 [WARN] agent: Join LAN failed: , retrying in 30s
==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=ea877ba1-2fdf-2b1e-0d01-a1815ca3b7c8&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2021/05/09 22:39:35 [ERR] agent: failed to sync remote state: No cluster leader
2021/05/09 22:39:37 [WARN] raft: no known peers, aborting election
2021/05/09 22:39:59 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/05/09 22:39:59 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
2021/05/09 22:39:59 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/05/09 22:39:59 [INFO] agent: (LAN) joined: 1 Err:
2021/05/09 22:39:59 [INFO] agent: Join LAN completed. Synced with 1 initial agents
2021/05/09 22:40:02 [ERR] agent: Coordinate update error: No cluster leader
2021/05/09 22:40:04 [ERR] agent: failed to sync remote state: No cluster leader
2021/05/09 22:40:06 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.210.71
2021/05/09 22:40:06 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.210.71:8300) (DC: dc1)
2021/05/09 22:40:06 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.210.71
2021/05/09 22:40:06 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/09 22:40:13 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 22:40:24 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/09 22:40:24 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 22:40:24 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.242.210.71:8300,10.242.163.157:8300,10.242.61.70:8300
2021/05/09 22:40:24 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.163.157
2021/05/09 22:40:24 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/05/09 22:40:25 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/05/09 22:40:25 [INFO] raft: Node at 10.242.61.70:8300 [Candidate] entering Candidate state in term 2
2021/05/09 22:40:25 [INFO] raft: Election won. Tally: 2
2021/05/09 22:40:25 [INFO] raft: Node at 10.242.61.70:8300 [Leader] entering Leader state
2021/05/09 22:40:25 [INFO] raft: Added peer b3d9ccb9-04c2-7b7e-2bea-552e392b6c00, starting replication
2021/05/09 22:40:25 [INFO] raft: Added peer 19ce88fc-998f-ee2b-b2d5-6fafec370e52, starting replication
2021/05/09 22:40:25 [INFO] consul: cluster leadership acquired
2021/05/09 22:40:25 [INFO] consul: New leader elected: dev-consul-server-0
2021/05/09 22:40:25 [WARN] raft: AppendEntries to {Voter b3d9ccb9-04c2-7b7e-2bea-552e392b6c00 10.242.210.71:8300} rejected, sending older logs (next: 1)
2021/05/09 22:40:25 [INFO] raft: pipelining replication to peer {Voter b3d9ccb9-04c2-7b7e-2bea-552e392b6c00 10.242.210.71:8300}
2021/05/09 22:40:25 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/09 22:40:25 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/09 22:40:25 [INFO] raft: pipelining replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/09 22:40:25 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/05/09 22:40:25 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/09 22:40:25 [INFO] agent: Synced node info
2021/05/09 22:40:40 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:40:40 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 22:40:54 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:40:56 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:40:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:40:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:41:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:41:04 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:41:04 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/09 22:41:16 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:41:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:41:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:41:40 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:41:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:41:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:41:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:41:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:41:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:43:10 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:43:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:43:12 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 22:43:16 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:43:52 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:44:40 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:44:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:44:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:45:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:45:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:45:32 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:46:30 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:46:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:46:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:46:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:46:56 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:46:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:46:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:47:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:47:04 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:47:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:47:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:47:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:47:36 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:47:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:47:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:47:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:47:44 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:47:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:47:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:47:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:47:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:47:54 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:47:54 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 22:47:54 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/09 22:47:56 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:48:02 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:48:02 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 22:48:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:48:10 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:48:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:48:26 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/09 22:48:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:48:36 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:48:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:49:06 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:49:06 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/09 22:49:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:49:16 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:49:16 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 22:49:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:49:56 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:49:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:50:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:50:22 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:50:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:50:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:51:06 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:51:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:51:16 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:51:16 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 22:51:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:51:22 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:52:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:52:26 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 22:54:18 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:54:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:54:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:54:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:54:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:54:32 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:54:32 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/09 22:54:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:54:36 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:54:38 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:54:38 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/09 22:54:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:54:52 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:54:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:55:32 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:56:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:57:22 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:58:10 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:58:10 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/09 22:58:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 22:58:42 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:58:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 22:58:58 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 22:58:58 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/09 23:02:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:02:58 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 23:02:58 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/09 23:05:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:05:18 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:05:18 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 23:05:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:25:00 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:25:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:25:10 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:25:10 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 23:25:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:54498->10.242.163.138:8301: i/o timeout
2021/05/09 23:25:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:25:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:44910->10.242.163.157:8301: i/o timeout
2021/05/09 23:25:12 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:25:14 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:25:14 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 23:25:14 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/09 23:25:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:25:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:44938->10.242.163.157:8301: i/o timeout
2021/05/09 23:25:15 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:25:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:54520->10.242.163.138:8301: i/o timeout
2021/05/09 23:25:17 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500178572s
2021/05/09 23:25:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:25:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:54550->10.242.163.138:8301: i/o timeout
2021/05/09 23:25:18 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:25:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:25:20 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.50017638s
2021/05/09 23:25:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:44982->10.242.163.157:8301: i/o timeout
2021/05/09 23:25:20 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:25:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:25:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:54592->10.242.163.138:8301: i/o timeout
2021/05/09 23:25:22 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:25:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:25:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33192->10.242.163.157:8302: i/o timeout
2021/05/09 23:25:23 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:25:24 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/05/09 23:25:24 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/09 23:25:24 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:25:24 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/05/09 23:25:24 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/09 23:25:24 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:25:24 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/09 23:25:24 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/05/09 23:25:26 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:25:26 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/09 23:25:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:25:26 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-xplxm but other probes failed, network may be misconfigured
2021/05/09 23:25:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:25:30 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:25:30 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 23:25:30 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/09 23:25:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:45074->10.242.163.157:8301: i/o timeout
2021/05/09 23:25:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:25:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:25:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:45108->10.242.163.157:8301: i/o timeout
2021/05/09 23:25:34 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:25:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:25:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:45116->10.242.163.157:8301: i/o timeout
2021/05/09 23:25:36 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:25:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:25:37 [ERR] raft: Failed to pipeline AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: write tcp 10.242.61.70:37834->10.242.163.157:8300: use of closed network connection
2021/05/09 23:25:37 [INFO] raft: aborting pipeline replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/09 23:25:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:45142->10.242.163.157:8301: i/o timeout
2021/05/09 23:25:38 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:25:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:25:40 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500280468s
2021/05/09 23:25:40 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:25:40 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/09 23:25:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:25:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:54766->10.242.163.138:8301: i/o timeout
2021/05/09 23:25:42 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:25:44 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/05/09 23:25:44 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:25:45 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/05/09 23:25:45 [INFO] raft: pipelining replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/09 23:25:46 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:25:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:25:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:25:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:45238->10.242.163.157:8301: i/o timeout
2021/05/09 23:25:49 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:25:50 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500200031s
2021/05/09 23:25:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:25:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:54846->10.242.163.138:8301: i/o timeout
2021/05/09 23:25:52 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:25:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:25:54 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500233365s
2021/05/09 23:25:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:45274->10.242.163.157:8301: i/o timeout
2021/05/09 23:25:54 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:25:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:25:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:54884->10.242.163.138:8301: i/o timeout
2021/05/09 23:25:56 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:25:57 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.923157035s
2021/05/09 23:25:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:25:57 [INFO] raft: aborting pipeline replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/09 23:25:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33496->10.242.163.157:8302: i/o timeout
2021/05/09 23:25:58 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:25:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:54926->10.242.163.138:8301: i/o timeout
2021/05/09 23:25:58 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:26:00 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500113983s
2021/05/09 23:26:00 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:26:00 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/09 23:26:00 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:26:00 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/09 23:26:01 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/05/09 23:26:01 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/09 23:26:01 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:26:01 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/05/09 23:26:01 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/09 23:26:01 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:26:01 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/09 23:26:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:45372->10.242.163.157:8301: i/o timeout
2021/05/09 23:26:03 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:26:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:07 [INFO] raft: pipelining replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/09 23:26:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:45422->10.242.163.157:8301: i/o timeout
2021/05/09 23:26:09 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:26:10 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:26:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:45442->10.242.163.157:8301: i/o timeout
2021/05/09 23:26:12 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:26:13 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500290347s
2021/05/09 23:26:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:45462->10.242.163.157:8301: i/o timeout
2021/05/09 23:26:14 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:26:15 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.993764887s
2021/05/09 23:26:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:45470->10.242.163.157:8301: i/o timeout
2021/05/09 23:26:16 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:26:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:45496->10.242.163.157:8301: i/o timeout
2021/05/09 23:26:18 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:26:18 [INFO] raft: aborting pipeline replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/09 23:26:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:20 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/05/09 23:26:21 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:26:21 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/09 23:26:22 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:26:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:45528->10.242.163.157:8301: i/o timeout
2021/05/09 23:26:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:45548->10.242.163.157:8301: i/o timeout
2021/05/09 23:26:23 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:26:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:25 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500282992s
2021/05/09 23:26:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:55152->10.242.163.138:8301: i/o timeout
2021/05/09 23:26:26 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:26:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:28 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.933944103s
2021/05/09 23:26:28 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.61.70:54524->10.242.163.157:8300: i/o timeout
2021/05/09 23:26:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:55172->10.242.163.138:8301: i/o timeout
2021/05/09 23:26:29 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:26:30 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 7.388209892s
2021/05/09 23:26:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:32 [INFO] raft: pipelining replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/09 23:26:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:45608->10.242.163.157:8301: i/o timeout
2021/05/09 23:26:32 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:26:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33806->10.242.163.157:8302: i/o timeout
2021/05/09 23:26:34 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:26:34 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/09 23:26:35 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/05/09 23:26:35 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/09 23:26:35 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:26:35 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/05/09 23:26:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:45644->10.242.163.157:8301: i/o timeout
2021/05/09 23:26:35 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:26:35 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500339053s
2021/05/09 23:26:36 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:26:37 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:26:38 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:26:39 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500184423s
2021/05/09 23:26:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:42 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.973130676s
2021/05/09 23:26:42 [INFO] raft: aborting pipeline replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/09 23:26:42 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:26:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33884->10.242.163.157:8302: i/o timeout
2021/05/09 23:26:43 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:26:45 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.50027293s
2021/05/09 23:26:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:47 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.94507679s
2021/05/09 23:26:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:47 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: read tcp 10.242.61.70:37832->10.242.163.157:8300: i/o timeout
2021/05/09 23:26:48 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:26:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:50 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 7.380354706s
2021/05/09 23:26:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:52 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.61.70:54728->10.242.163.157:8300: i/o timeout
2021/05/09 23:26:52 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:26:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:56 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/09 23:26:56 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.163.157
2021/05/09 23:26:56 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/05/09 23:26:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:26:58 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: read tcp 10.242.61.70:54776->10.242.163.157:8300: i/o timeout
2021/05/09 23:26:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34016->10.242.163.157:8302: i/o timeout
2021/05/09 23:26:58 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:26:58 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.163.157:8301
2021/05/09 23:26:59 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:26:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:27:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:27:01 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/09 23:27:01 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.163.138:49858
2021/05/09 23:27:02 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.61.70:54808->10.242.163.157:8300: i/o timeout
2021/05/09 23:27:02 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:27:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:27:03 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.163.157
2021/05/09 23:27:03 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/05/09 23:27:04 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:27:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:27:06 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500286957s
2021/05/09 23:27:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:27:08 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.999646509s
2021/05/09 23:27:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:27:11 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 7.410296224s
2021/05/09 23:27:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:27:12 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.61.70:54900->10.242.163.157:8300: i/o timeout
2021/05/09 23:27:12 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:27:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:27:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34138->10.242.163.157:8302: i/o timeout
2021/05/09 23:27:13 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:27:14 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: read tcp 10.242.61.70:54872->10.242.163.157:8300: i/o timeout
2021/05/09 23:27:15 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:27:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:27:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:27:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:27:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:27:22 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.61.70:54982->10.242.163.157:8300: i/o timeout
2021/05/09 23:27:22 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:27:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:27:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34230->10.242.163.157:8302: i/o timeout
2021/05/09 23:27:23 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:27:25 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: read tcp 10.242.61.70:54996->10.242.163.157:8300: i/o timeout
2021/05/09 23:27:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:27:25 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:27:25 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/09 23:27:25 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:27:26 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/05/09 23:27:26 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:27:26 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/09 23:27:26 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/09 23:27:26 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:27:26 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/09 23:27:28 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:27:28 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 23:27:28 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/09 23:27:29 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500229727s
2021/05/09 23:27:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:27:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:46106->10.242.163.157:8301: i/o timeout
2021/05/09 23:27:29 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:27:31 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.987136684s
2021/05/09 23:27:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:27:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:46114->10.242.163.157:8301: i/o timeout
2021/05/09 23:27:31 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:27:32 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.61.70:55068->10.242.163.157:8300: i/o timeout
2021/05/09 23:27:33 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/05/09 23:27:33 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:27:33 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/09 23:27:33 [INFO] raft: pipelining replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/09 23:28:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:28:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:46422->10.242.163.157:8301: i/o timeout
2021/05/09 23:28:08 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:28:09 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.501399961s
2021/05/09 23:29:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:56498->10.242.163.138:8301: i/o timeout
2021/05/09 23:29:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:30:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:56964->10.242.163.138:8301: i/o timeout
2021/05/09 23:30:06 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:30:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:57124->10.242.163.138:8301: i/o timeout
2021/05/09 23:30:25 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:30:48 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:30:48 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 23:30:48 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/09 23:31:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:32:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:32:08 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-xplxm but other probes failed, network may be misconfigured
2021/05/09 23:32:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:57958->10.242.163.138:8301: i/o timeout
2021/05/09 23:32:11 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:32:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:32:12 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 23:32:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:32:14 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500205738s
2021/05/09 23:32:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:57964->10.242.163.138:8301: i/o timeout
2021/05/09 23:32:14 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:32:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:58038->10.242.163.138:8301: i/o timeout
2021/05/09 23:32:20 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:32:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:32:50 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/09 23:32:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:32:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:58280->10.242.163.138:8301: i/o timeout
2021/05/09 23:32:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:33:07 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-xplxm)
2021/05/09 23:34:56 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:34:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:35:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:35:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:59670->10.242.163.138:8301: i/o timeout
2021/05/09 23:35:48 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:35:50 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/05/09 23:35:52 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:36:00 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:36:00 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 23:36:00 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/09 23:36:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:59788->10.242.163.138:8301: i/o timeout
2021/05/09 23:36:00 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:36:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:36:03 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:36:03 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/09 23:36:04 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-xplxm but other probes failed, network may be misconfigured
2021/05/09 23:36:04 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:36:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:36:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:50238->10.242.163.157:8301: i/o timeout
2021/05/09 23:36:07 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500199262s
2021/05/09 23:36:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:36:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:36:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:50278->10.242.163.157:8301: i/o timeout
2021/05/09 23:36:10 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:36:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:36:12 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 23:36:12 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/09 23:36:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:50284->10.242.163.157:8301: i/o timeout
2021/05/09 23:36:12 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:36:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:36:15 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:36:15 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/09 23:36:15 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-xplxm)
2021/05/09 23:36:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:36:22 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:36:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:36:26 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 23:36:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:40:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:41:10 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:41:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33998->10.242.163.138:8301: i/o timeout
2021/05/09 23:41:10 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:41:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34082->10.242.163.138:8301: i/o timeout
2021/05/09 23:41:21 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:41:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34094->10.242.163.138:8301: i/o timeout
2021/05/09 23:41:23 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:41:24 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (2 peer confirmations)
2021/05/09 23:41:24 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:41:24 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/09 23:41:25 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:41:25 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/09 23:41:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34158->10.242.163.138:8301: i/o timeout
2021/05/09 23:41:30 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:41:34 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (2 peer confirmations)
2021/05/09 23:41:34 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:41:34 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/09 23:41:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34192->10.242.163.138:8301: i/o timeout
2021/05/09 23:41:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:41:35 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:41:35 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/09 23:41:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34212->10.242.163.138:8301: i/o timeout
2021/05/09 23:41:37 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:41:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34222->10.242.163.138:8301: i/o timeout
2021/05/09 23:41:40 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:41:41 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (2 peer confirmations)
2021/05/09 23:41:41 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:41:41 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/09 23:41:50 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:41:50 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/09 23:41:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:52990->10.242.163.157:8301: i/o timeout
2021/05/09 23:41:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:41:56 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:41:56 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 23:41:56 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/09 23:41:57 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500253332s
2021/05/09 23:41:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:42:06 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:42:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:42:12 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 23:42:12 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/09 23:42:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34782->10.242.163.138:8301: i/o timeout
2021/05/09 23:42:48 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:43:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:35052->10.242.163.138:8301: i/o timeout
2021/05/09 23:43:21 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:44:52 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500186221s
2021/05/09 23:44:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:54400->10.242.163.157:8301: i/o timeout
2021/05/09 23:44:53 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:44:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:44:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:35830->10.242.163.138:8301: i/o timeout
2021/05/09 23:45:00 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:45:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:35840->10.242.163.138:8301: i/o timeout
2021/05/09 23:45:03 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:45:03 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (2 peer confirmations)
2021/05/09 23:45:03 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:45:03 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/09 23:45:03 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:45:03 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/09 23:45:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:54522->10.242.163.157:8301: i/o timeout
2021/05/09 23:45:07 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:45:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:35916->10.242.163.138:8301: i/o timeout
2021/05/09 23:45:10 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:45:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:45:13 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.50017926s
2021/05/09 23:45:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:35924->10.242.163.138:8301: i/o timeout
2021/05/09 23:45:13 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:45:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:54588->10.242.163.157:8301: i/o timeout
2021/05/09 23:45:16 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:45:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:54606->10.242.163.157:8301: i/o timeout
2021/05/09 23:45:18 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:45:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:36004->10.242.163.138:8301: i/o timeout
2021/05/09 23:45:20 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:45:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:22 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (1 peer confirmations)
2021/05/09 23:45:22 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:45:22 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/09 23:45:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:54646->10.242.163.157:8301: i/o timeout
2021/05/09 23:45:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:45:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:42842->10.242.163.157:8302: i/o timeout
2021/05/09 23:45:23 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:45:24 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.50019806s
2021/05/09 23:45:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:54686->10.242.163.157:8301: i/o timeout
2021/05/09 23:45:25 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:45:25 [ERR] memberlist: failed to receive: read tcp 10.242.61.70:8301->10.242.163.157:56914: i/o timeout from=10.242.163.157:56914
2021/05/09 23:45:25 [ERR] memberlist: Failed to send error: write tcp 10.242.61.70:8301->10.242.163.157:56914: i/o timeout from=10.242.163.157:56914
2021/05/09 23:45:26 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.996020621s
2021/05/09 23:45:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:54694->10.242.163.157:8301: i/o timeout
2021/05/09 23:45:27 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:45:28 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/05/09 23:45:28 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/09 23:45:28 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:45:28 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/05/09 23:45:29 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 7.441618276s
2021/05/09 23:45:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:31 [ERR] raft: Failed to pipeline AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: write tcp 10.242.61.70:55152->10.242.163.157:8300: use of closed network connection
2021/05/09 23:45:31 [INFO] raft: aborting pipeline replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/09 23:45:31 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:45:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:32 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: read tcp 10.242.61.70:55092->10.242.163.157:8300: i/o timeout
2021/05/09 23:45:33 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:45:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:42922->10.242.163.157:8302: i/o timeout
2021/05/09 23:45:33 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:45:33 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.50027383s
2021/05/09 23:45:34 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:45:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:36 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.949990474s
2021/05/09 23:45:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:38 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.163.157:8301
2021/05/09 23:45:38 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 7.423393269s
2021/05/09 23:45:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:41 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.61.70:35516->10.242.163.157:8300: i/o timeout
2021/05/09 23:45:41 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:45:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:43 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: read tcp 10.242.61.70:35532->10.242.163.157:8300: i/o timeout
2021/05/09 23:45:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:44 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:45:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:43004->10.242.163.157:8302: i/o timeout
2021/05/09 23:45:48 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:45:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:50 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/05/09 23:45:51 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.61.70:35598->10.242.163.157:8300: i/o timeout
2021/05/09 23:45:51 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:45:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:53 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/09 23:45:53 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.163.157
2021/05/09 23:45:53 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/05/09 23:45:54 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: read tcp 10.242.61.70:35626->10.242.163.157:8300: i/o timeout
2021/05/09 23:45:54 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:45:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:45:57 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:45:57 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/09 23:45:58 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:45:58 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.163.157:8302
2021/05/09 23:45:59 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.163.157
2021/05/09 23:45:59 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/05/09 23:45:59 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/09 23:45:59 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:45:59 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/09 23:45:59 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/05/09 23:45:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:46:00 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500199133s
2021/05/09 23:46:01 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.61.70:35684->10.242.163.157:8300: i/o timeout
2021/05/09 23:46:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:46:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:36362->10.242.163.138:8301: i/o timeout
2021/05/09 23:46:02 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:46:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:46:03 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500237595s
2021/05/09 23:46:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:46:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:55022->10.242.163.157:8301: i/o timeout
2021/05/09 23:46:05 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:46:06 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.929510429s
2021/05/09 23:46:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:46:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:36396->10.242.163.138:8301: i/o timeout
2021/05/09 23:46:07 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:46:08 [WARN] raft: Rejecting vote request from 10.242.163.157:8300 since we have a leader: 10.242.61.70:8300
2021/05/09 23:46:08 [DEBUG] raft-net: 10.242.61.70:8300 accepted connection from: 10.242.163.157:49714
2021/05/09 23:46:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:43226->10.242.163.157:8302: i/o timeout
2021/05/09 23:46:08 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:46:08 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 7.424619024s
2021/05/09 23:46:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:46:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:36416->10.242.163.138:8301: i/o timeout
2021/05/09 23:46:09 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:46:11 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (1 peer confirmations)
2021/05/09 23:46:11 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:46:11 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/09 23:46:11 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.61.70:35712->10.242.163.157:8300: i/o timeout
2021/05/09 23:46:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:46:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:55074->10.242.163.157:8301: i/o timeout
2021/05/09 23:46:11 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:46:12 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: read tcp 10.242.61.70:35780->10.242.163.157:8300: i/o timeout
2021/05/09 23:46:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:46:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:55098->10.242.163.157:8301: i/o timeout
2021/05/09 23:46:13 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:46:15 [ERR] memberlist: Push/Pull with dev-consul-server-2 failed: read tcp 10.242.61.70:55030->10.242.163.157:8301: i/o timeout
2021/05/09 23:46:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:46:18 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:46:19 [INFO] raft: pipelining replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/09 23:46:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:46:20 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:46:20 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/09 23:46:22 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:46:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:46:28 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:46:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:46:32 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:46:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:46:33 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500155766s
2021/05/09 23:46:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:36634->10.242.163.138:8301: i/o timeout
2021/05/09 23:46:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:46:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:47:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:47:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:47:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:37246->10.242.163.138:8301: i/o timeout
2021/05/09 23:47:53 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:47:54 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:47:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:47:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:55902->10.242.163.157:8301: i/o timeout
2021/05/09 23:47:56 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:47:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:47:57 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (2 peer confirmations)
2021/05/09 23:47:57 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:47:57 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/09 23:47:58 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500228287s
2021/05/09 23:47:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:37286->10.242.163.138:8301: i/o timeout
2021/05/09 23:47:58 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:47:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:48:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:55952->10.242.163.157:8301: i/o timeout
2021/05/09 23:48:00 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:48:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:48:02 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/09 23:48:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:48:04 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/09 23:48:04 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:48:04 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/09 23:48:05 [ERR] raft: Failed to pipeline AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: write tcp 10.242.61.70:35864->10.242.163.157:8300: use of closed network connection
2021/05/09 23:48:05 [INFO] raft: aborting pipeline replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/09 23:48:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:48:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:48:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:37372->10.242.163.138:8301: i/o timeout
2021/05/09 23:48:07 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:48:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:48:10 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500259711s
2021/05/09 23:48:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:56018->10.242.163.157:8301: i/o timeout
2021/05/09 23:48:10 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:48:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:48:12 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.991046291s
2021/05/09 23:48:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:48:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:56042->10.242.163.157:8301: i/o timeout
2021/05/09 23:48:13 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:48:15 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 7.45168605s
2021/05/09 23:48:15 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.61.70:36782->10.242.163.157:8300: i/o timeout
2021/05/09 23:48:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:48:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:37438->10.242.163.138:8301: i/o timeout
2021/05/09 23:48:16 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:48:17 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/09 23:48:17 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/05/09 23:48:18 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/09 23:48:18 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.163.157:42900
2021/05/09 23:48:18 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: read tcp 10.242.61.70:35878->10.242.163.157:8300: i/o timeout
2021/05/09 23:48:18 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:48:18 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/09 23:48:18 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:48:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:44278->10.242.163.157:8302: i/o timeout
2021/05/09 23:48:18 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:48:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:56110->10.242.163.157:8301: i/o timeout
2021/05/09 23:48:18 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:48:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:48:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:56130->10.242.163.157:8301: i/o timeout
2021/05/09 23:48:20 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:48:21 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/09 23:48:21 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:48:21 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/05/09 23:48:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:48:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:48:25 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.61.70:36866->10.242.163.157:8300: i/o timeout
2021/05/09 23:48:25 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:48:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:48:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:48:28 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:48:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:48:30 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500174643s
2021/05/09 23:48:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:48:32 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.918581969s
2021/05/09 23:48:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:48:35 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 7.359815027s
2021/05/09 23:48:35 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.61.70:36962->10.242.163.157:8300: i/o timeout
2021/05/09 23:48:35 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:48:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:48:37 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:48:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:48:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:44454->10.242.163.157:8302: i/o timeout
2021/05/09 23:48:38 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:48:39 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500152729s
2021/05/09 23:48:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:48:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:48:41 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:48:43 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.50017215s
2021/05/09 23:48:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:48:44 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:48:44 [INFO] raft: pipelining replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/09 23:48:44 [ERR] memberlist: failed to receive: read tcp 10.242.61.70:8301->10.242.163.138:41234: i/o timeout from=10.242.163.138:41234
2021/05/09 23:48:44 [ERR] memberlist: Failed to send error: write tcp 10.242.61.70:8301->10.242.163.138:41234: i/o timeout from=10.242.163.138:41234
2021/05/09 23:48:44 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.163.157:8301
2021/05/09 23:48:44 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:48:45 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:48:46 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:48:47 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500265089s
2021/05/09 23:48:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:48:49 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:48:49 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:48:49 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/09 23:48:50 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:48:50 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/09 23:48:50 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:48:50 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/09 23:48:50 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:48:50 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/09 23:48:52 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:48:52 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/09 23:48:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:37760->10.242.163.138:8301: i/o timeout
2021/05/09 23:48:53 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:48:54 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.50017164s
2021/05/09 23:48:54 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:48:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:37812->10.242.163.138:8301: i/o timeout
2021/05/09 23:48:58 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:49:01 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/09 23:49:02 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:49:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:37842->10.242.163.138:8301: i/o timeout
2021/05/09 23:49:02 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:49:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:49:05 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-xplxm but other probes failed, network may be misconfigured
2021/05/09 23:49:06 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:49:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:49:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:56520->10.242.163.157:8301: i/o timeout
2021/05/09 23:49:07 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:49:07 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500265833s
2021/05/09 23:49:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:49:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:56538->10.242.163.157:8301: i/o timeout
2021/05/09 23:49:09 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:49:10 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (2 peer confirmations)
2021/05/09 23:49:10 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:49:10 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/09 23:49:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:37916->10.242.163.138:8301: i/o timeout
2021/05/09 23:49:11 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:49:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:49:13 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500213108s
2021/05/09 23:49:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:49:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:56572->10.242.163.157:8301: i/o timeout
2021/05/09 23:49:13 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:49:13 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:49:13 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/09 23:49:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:49:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:37952->10.242.163.138:8301: i/o timeout
2021/05/09 23:49:15 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:49:16 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.979498065s
2021/05/09 23:49:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:49:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:56598->10.242.163.157:8301: i/o timeout
2021/05/09 23:49:17 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:49:18 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 7.438736573s
2021/05/09 23:49:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:49:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:37982->10.242.163.138:8301: i/o timeout
2021/05/09 23:49:19 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:49:21 [ERR] raft: Failed to pipeline AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: write tcp 10.242.61.70:37040->10.242.163.157:8300: use of closed network connection
2021/05/09 23:49:21 [INFO] raft: aborting pipeline replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/09 23:49:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:49:21 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/09 23:49:22 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/09 23:49:22 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.163.157:52892
2021/05/09 23:49:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:49:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:44844->10.242.163.157:8302: i/o timeout
2021/05/09 23:49:23 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:49:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:38034->10.242.163.138:8301: i/o timeout
2021/05/09 23:49:23 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:49:24 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500165168s
2021/05/09 23:49:25 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/05/09 23:49:25 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/09 23:49:25 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:49:25 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/05/09 23:49:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:49:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:56678->10.242.163.157:8301: i/o timeout
2021/05/09 23:49:25 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:49:27 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (1 peer confirmations)
2021/05/09 23:49:27 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:49:27 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/09 23:49:27 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:49:27 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/09 23:49:27 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:49:27 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/09 23:49:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:49:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:38064->10.242.163.138:8301: i/o timeout
2021/05/09 23:49:27 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:49:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:49:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:56724->10.242.163.157:8301: i/o timeout
2021/05/09 23:49:29 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:49:29 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:49:29 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/09 23:49:29 [INFO] raft: pipelining replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/09 23:49:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:38218->10.242.163.138:8301: i/o timeout
2021/05/09 23:49:46 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:50:38 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:50:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:50:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:57268->10.242.163.157:8301: i/o timeout
2021/05/09 23:50:39 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:50:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:38666->10.242.163.138:8301: i/o timeout
2021/05/09 23:50:43 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:50:44 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:50:44 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500348105s
2021/05/09 23:50:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:50:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:38696->10.242.163.138:8301: i/o timeout
2021/05/09 23:50:47 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:51:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:51:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:39268->10.242.163.138:8301: i/o timeout
2021/05/09 23:51:58 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:51:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:56:30 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:56:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:56:34 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:56:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:41510->10.242.163.138:8301: i/o timeout
2021/05/09 23:56:45 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:56:46 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:56:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:56:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:41564->10.242.163.138:8301: i/o timeout
2021/05/09 23:56:52 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:56:56 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:56:56 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (2 peer confirmations)
2021/05/09 23:56:56 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:56:56 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/09 23:56:56 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:56:56 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/09 23:56:57 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-xplxm)
2021/05/09 23:56:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:57:00 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:57:06 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:57:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:57:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:41698->10.242.163.138:8301: i/o timeout
2021/05/09 23:57:08 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:57:09 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-xplxm)
2021/05/09 23:57:20 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:57:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:42030->10.242.163.138:8301: i/o timeout
2021/05/09 23:57:50 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:58:04 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:58:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:58:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:42156->10.242.163.138:8301: i/o timeout
2021/05/09 23:58:06 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:58:10 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:58:10 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 23:58:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:58:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:58:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:42212->10.242.163.138:8301: i/o timeout
2021/05/09 23:58:13 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:58:15 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500263178s
2021/05/09 23:58:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:58:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:42242->10.242.163.138:8301: i/o timeout
2021/05/09 23:58:17 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:58:17 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (2 peer confirmations)
2021/05/09 23:58:17 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:58:17 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/09 23:58:18 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:58:18 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/09 23:58:18 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:58:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:58:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:42272->10.242.163.138:8301: i/o timeout
2021/05/09 23:58:19 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:58:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:58:21 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500197139s
2021/05/09 23:58:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:60920->10.242.163.157:8301: i/o timeout
2021/05/09 23:58:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:58:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:58:24 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.992086669s
2021/05/09 23:58:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:42312->10.242.163.138:8301: i/o timeout
2021/05/09 23:58:24 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:58:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:58:26 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 23:58:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:60962->10.242.163.157:8301: i/o timeout
2021/05/09 23:58:26 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 7.468386474s
2021/05/09 23:58:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:58:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:42334->10.242.163.138:8301: i/o timeout
2021/05/09 23:58:27 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:58:28 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:58:28 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/09 23:58:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:58:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:58:32 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:58:32 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-xplxm)
2021/05/09 23:58:32 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/09 23:58:34 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:58:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:58:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:32806->10.242.163.157:8301: i/o timeout
2021/05/09 23:58:35 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:58:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:58:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:42406->10.242.163.138:8301: i/o timeout
2021/05/09 23:58:38 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:58:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:32840->10.242.163.157:8301: i/o timeout
2021/05/09 23:58:41 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:58:42 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500367741s
2021/05/09 23:58:42 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:58:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:58:44 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-xplxm but other probes failed, network may be misconfigured
2021/05/09 23:58:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:58:46 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (2 peer confirmations)
2021/05/09 23:58:46 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:58:46 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/09 23:58:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:32898->10.242.163.157:8301: i/o timeout
2021/05/09 23:58:46 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:58:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:58:48 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500230334s
2021/05/09 23:58:48 [ERR] raft: Failed to pipeline AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: write tcp 10.242.61.70:37430->10.242.163.157:8300: use of closed network connection
2021/05/09 23:58:48 [INFO] raft: aborting pipeline replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/09 23:58:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:32914->10.242.163.157:8301: i/o timeout
2021/05/09 23:58:48 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:58:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:58:49 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-xplxm 10.242.163.138:8301
2021/05/09 23:58:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:58:52 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500309073s
2021/05/09 23:58:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:32956->10.242.163.157:8301: i/o timeout
2021/05/09 23:58:52 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:58:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:58:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:49370->10.242.163.157:8302: i/o timeout
2021/05/09 23:58:53 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:58:54 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.933263877s
2021/05/09 23:58:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:32982->10.242.163.157:8301: i/o timeout
2021/05/09 23:58:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:58:56 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:58:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:58:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33004->10.242.163.157:8301: i/o timeout
2021/05/09 23:58:57 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:58:58 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.61.70:41940->10.242.163.157:8300: i/o timeout
2021/05/09 23:58:58 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500207256s
2021/05/09 23:58:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:58:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33032->10.242.163.157:8301: i/o timeout
2021/05/09 23:58:59 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:00 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/05/09 23:59:00 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/09 23:59:00 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:59:00 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/05/09 23:59:01 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.977014373s
2021/05/09 23:59:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:59:02 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/05/09 23:59:03 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:59:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:59:04 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:59:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:59:06 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500412075s
2021/05/09 23:59:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:59:08 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.61.70:42040->10.242.163.157:8300: i/o timeout
2021/05/09 23:59:08 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:59:09 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.969471335s
2021/05/09 23:59:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:59:09 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/09 23:59:09 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:59:09 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/09 23:59:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:59:12 [ERR] raft: peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300} has newer term, stopping replication
2021/05/09 23:59:12 [INFO] raft: Node at 10.242.61.70:8300 [Follower] entering Follower state (Leader: "")
2021/05/09 23:59:12 [INFO] raft: aborting pipeline replication to peer {Voter b3d9ccb9-04c2-7b7e-2bea-552e392b6c00 10.242.210.71:8300}
2021/05/09 23:59:12 [INFO] consul: cluster leadership lost
2021/05/09 23:59:12 [WARN] raft: Rejecting vote request from 10.242.163.157:8300 since our last index is greater (1718, 1704)
2021/05/09 23:59:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33148->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:14 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33162->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:16 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33174->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:18 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33210->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:20 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:21 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.245.140:49980
2021/05/09 23:59:21 [WARN] raft: Rejecting vote request from 10.242.163.157:8300 since our last index is greater (1718, 1704)
2021/05/09 23:59:21 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/05/09 23:59:21 [INFO] raft: Node at 10.242.61.70:8300 [Candidate] entering Candidate state in term 5
2021/05/09 23:59:21 [INFO] raft: Election won. Tally: 2
2021/05/09 23:59:21 [INFO] raft: Node at 10.242.61.70:8300 [Leader] entering Leader state
2021/05/09 23:59:21 [INFO] raft: Added peer b3d9ccb9-04c2-7b7e-2bea-552e392b6c00, starting replication
2021/05/09 23:59:21 [INFO] raft: Added peer 19ce88fc-998f-ee2b-b2d5-6fafec370e52, starting replication
2021/05/09 23:59:21 [INFO] consul: cluster leadership acquired
2021/05/09 23:59:21 [INFO] consul: New leader elected: dev-consul-server-0
2021/05/09 23:59:21 [INFO] raft: pipelining replication to peer {Voter b3d9ccb9-04c2-7b7e-2bea-552e392b6c00 10.242.210.71:8300}
2021/05/09 23:59:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33226->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:24 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.501938507s
2021/05/09 23:59:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33246->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:24 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:24 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:59:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:59:26 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:59:26 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/09 23:59:26 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/09 23:59:26 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.969688228s
2021/05/09 23:59:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:59:28 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/09 23:59:29 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 7.410382239s
2021/05/09 23:59:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33296->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:29 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:59:31 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-xplxm but other probes failed, network may be misconfigured
2021/05/09 23:59:31 [ERR] raft: Failed to make RequestVote RPC to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.61.70:36910->10.242.163.157:8300: i/o timeout
2021/05/09 23:59:31 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.61.70:42120->10.242.163.157:8300: i/o timeout
2021/05/09 23:59:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:59:32 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: read tcp 10.242.61.70:42248->10.242.163.157:8300: i/o timeout
2021/05/09 23:59:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:42926->10.242.163.138:8301: i/o timeout
2021/05/09 23:59:33 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:59:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:59:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33346->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:35 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:59:37 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-xplxm but other probes failed, network may be misconfigured
2021/05/09 23:59:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:59:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:49784->10.242.163.157:8302: i/o timeout
2021/05/09 23:59:38 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:59:39 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (1 peer confirmations)
2021/05/09 23:59:39 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:59:39 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/09 23:59:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33390->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:39 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:59:41 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/05/09 23:59:41 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/09 23:59:41 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:59:41 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/05/09 23:59:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33404->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:41 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:59:41 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.61.70:42334->10.242.163.157:8300: i/o timeout
2021/05/09 23:59:42 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:59:43 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: read tcp 10.242.61.70:42350->10.242.163.157:8300: i/o timeout
2021/05/09 23:59:43 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:59:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:59:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:59:46 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/09 23:59:46 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/09 23:59:46 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:59:46 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/09 23:59:46 [WARN] raft: AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300} rejected, sending older logs (next: 1705)
2021/05/09 23:59:46 [INFO] raft: pipelining replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/09 23:59:47 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:59:47 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/09 23:59:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:59:48 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/09 23:59:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33484->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:49 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:50 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500269796s
2021/05/09 23:59:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/09 23:59:50 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/09 23:59:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:59:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:43088->10.242.163.138:8301: i/o timeout
2021/05/09 23:59:52 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:59:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:59:55 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500276958s
2021/05/09 23:59:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33518->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:59:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:43146->10.242.163.138:8301: i/o timeout
2021/05/09 23:59:57 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:59:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/09 23:59:57 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.942501849s
2021/05/09 23:59:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33568->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:59 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:00:00 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:00:00 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:00:00 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/05/10 00:00:01 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:00:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:43190->10.242.163.138:8301: i/o timeout
2021/05/10 00:00:01 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:00:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:00:02 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:00:03 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:00:03 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:00:03 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/10 00:00:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:43196->10.242.163.138:8301: i/o timeout
2021/05/10 00:00:03 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:00:04 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:00:04 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:00:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:00:06 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:00:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:00:08 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500211988s
2021/05/10 00:00:08 [ERR] raft: Failed to pipeline AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: write tcp 10.242.61.70:42428->10.242.163.157:8300: use of closed network connection
2021/05/10 00:00:08 [INFO] raft: aborting pipeline replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/10 00:00:08 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:00:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:00:11 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500272083s
2021/05/10 00:00:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:00:13 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/05/10 00:00:13 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:00:13 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/10 00:00:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:00:14 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:00:14 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:00:14 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:00:14 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/10 00:00:15 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/10 00:00:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:00:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33718->10.242.163.157:8301: i/o timeout
2021/05/10 00:00:17 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:00:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:00:18 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.61.70:42666->10.242.163.157:8300: i/o timeout
2021/05/10 00:00:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33746->10.242.163.157:8301: i/o timeout
2021/05/10 00:00:19 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:00:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:00:21 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500259594s
2021/05/10 00:00:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:00:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:43356->10.242.163.138:8301: i/o timeout
2021/05/10 00:00:22 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:00:23 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:00:23 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:00:23 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/05/10 00:00:23 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.96683745s
2021/05/10 00:00:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:00:24 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/10 00:00:25 [ERR] raft: peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300} has newer term, stopping replication
2021/05/10 00:00:25 [INFO] raft: Node at 10.242.61.70:8300 [Follower] entering Follower state (Leader: "")
2021/05/10 00:00:25 [INFO] raft: aborting pipeline replication to peer {Voter b3d9ccb9-04c2-7b7e-2bea-552e392b6c00 10.242.210.71:8300}
2021/05/10 00:00:25 [INFO] consul: cluster leadership lost
2021/05/10 00:00:25 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:00:25 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:00:26 [WARN] raft: Rejecting vote request from 10.242.163.157:8300 since our last index is greater (1752, 1737)
2021/05/10 00:00:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:43442->10.242.163.138:8301: i/o timeout
2021/05/10 00:00:31 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:00:33 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/10 00:00:34 [DEBUG] raft-net: 10.242.61.70:8300 accepted connection from: 10.242.210.71:37114
2021/05/10 00:00:34 [INFO] consul: New leader elected: dev-consul-server-1
2021/05/10 00:00:35 [DEBUG] raft-net: 10.242.61.70:8300 accepted connection from: 10.242.210.71:37134
2021/05/10 00:00:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33880->10.242.163.157:8301: i/o timeout
2021/05/10 00:00:35 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:00:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:43478->10.242.163.138:8301: i/o timeout
2021/05/10 00:00:37 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:00:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:43502->10.242.163.138:8301: i/o timeout
2021/05/10 00:00:39 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:00:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33930->10.242.163.157:8301: i/o timeout
2021/05/10 00:00:41 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:00:42 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/05/10 00:00:42 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:00:42 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:00:42 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:00:42 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:00:43 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (1 peer confirmations)
2021/05/10 00:00:43 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:00:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33938->10.242.163.157:8301: i/o timeout
2021/05/10 00:00:43 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:00:44 [WARN] raft: Rejecting vote request from 10.242.163.157:8300 since we have a leader: 10.242.210.71:8300
2021/05/10 00:00:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:33976->10.242.163.157:8301: i/o timeout
2021/05/10 00:00:46 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:00:49 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/10 00:00:54 [ERR] memberlist: failed to receive: read tcp 10.242.61.70:8301->10.242.163.138:52542: i/o timeout from=10.242.163.138:52542
2021/05/10 00:00:54 [ERR] memberlist: Failed to send error: write tcp 10.242.61.70:8301->10.242.163.138:52542: i/o timeout from=10.242.163.138:52542
2021/05/10 00:00:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34062->10.242.163.157:8301: i/o timeout
2021/05/10 00:00:57 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/10 00:00:58 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/05/10 00:00:58 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:01:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:43708->10.242.163.138:8301: i/o timeout
2021/05/10 00:01:01 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:01:04 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (2 peer confirmations)
2021/05/10 00:01:04 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:01:04 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-xplxm but other probes failed, network may be misconfigured
2021/05/10 00:01:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34188->10.242.163.157:8301: i/o timeout
2021/05/10 00:01:12 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:01:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:43808->10.242.163.138:8301: i/o timeout
2021/05/10 00:01:14 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:01:19 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-xplxm but other probes failed, network may be misconfigured
2021/05/10 00:01:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:43870->10.242.163.138:8301: i/o timeout
2021/05/10 00:01:21 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:01:23 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/10 00:01:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:43912->10.242.163.138:8301: i/o timeout
2021/05/10 00:01:25 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:01:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34322->10.242.163.157:8301: i/o timeout
2021/05/10 00:01:27 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:01:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34340->10.242.163.157:8301: i/o timeout
2021/05/10 00:01:29 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:01:30 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:01:30 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:01:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:43952->10.242.163.138:8301: i/o timeout
2021/05/10 00:01:31 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:01:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:43986->10.242.163.138:8301: i/o timeout
2021/05/10 00:01:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:01:36 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:01:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:43996->10.242.163.138:8301: i/o timeout
2021/05/10 00:01:36 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:01:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:50828->10.242.163.157:8302: i/o timeout
2021/05/10 00:01:38 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:01:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:50906->10.242.163.157:8302: i/o timeout
2021/05/10 00:01:48 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:01:51 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/10 00:01:56 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/05/10 00:01:56 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.163.157:8301
2021/05/10 00:01:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:50992->10.242.163.157:8302: i/o timeout
2021/05/10 00:01:58 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:01:59 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/05/10 00:02:05 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:02:05 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:02:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34652->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:06 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:02:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34672->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:09 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:02:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34700->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:12 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:02:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:51078->10.242.163.157:8302: i/o timeout
2021/05/10 00:02:13 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:02:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34714->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:14 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:02:17 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/05/10 00:02:17 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:02:17 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:02:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34742->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:17 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:02:17 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:02:18 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:02:18 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:02:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34776->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34800->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:23 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:02:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34822->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:26 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:02:27 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/05/10 00:02:27 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:02:27 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:02:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:51246->10.242.163.157:8302: i/o timeout
2021/05/10 00:02:28 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:02:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:44468->10.242.163.138:8301: i/o timeout
2021/05/10 00:02:32 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:02:33 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/05/10 00:02:33 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:02:33 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:02:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:44490->10.242.163.138:8301: i/o timeout
2021/05/10 00:02:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:02:38 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-xplxm but other probes failed, network may be misconfigured
2021/05/10 00:02:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34948->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:41 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:02:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:44566->10.242.163.138:8301: i/o timeout
2021/05/10 00:02:43 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:02:45 [ERR] memberlist: Push/Pull with dev-consul-7467bf7b46-xplxm failed: read tcp 10.242.61.70:44496->10.242.163.138:8301: i/o timeout
2021/05/10 00:02:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:34978->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:45 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:02:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:44592->10.242.163.138:8301: i/o timeout
2021/05/10 00:02:47 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:02:48 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (2 peer confirmations)
2021/05/10 00:02:48 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:02:48 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/05/10 00:02:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:35012->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:49 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:02:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:35036->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:51 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:02:53 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/05/10 00:02:53 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:02:53 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:02:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:51464->10.242.163.157:8302: i/o timeout
2021/05/10 00:02:53 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:02:58 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:02:58 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:03:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:35122->10.242.163.157:8301: i/o timeout
2021/05/10 00:03:00 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:03:01 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/05/10 00:03:01 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/05/10 00:03:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:35146->10.242.163.157:8301: i/o timeout
2021/05/10 00:03:04 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:03:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:35166->10.242.163.157:8301: i/o timeout
2021/05/10 00:03:06 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:03:06 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-xplxm 10.242.163.138:8301
2021/05/10 00:03:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:35188->10.242.163.157:8301: i/o timeout
2021/05/10 00:03:08 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:03:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:35210->10.242.163.157:8301: i/o timeout
2021/05/10 00:03:10 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:03:12 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/05/10 00:03:12 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:03:12 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:03:14 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:03:14 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:03:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:35250->10.242.163.157:8301: i/o timeout
2021/05/10 00:03:16 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:03:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:35286->10.242.163.157:8301: i/o timeout
2021/05/10 00:03:19 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:03:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:35308->10.242.163.157:8301: i/o timeout
2021/05/10 00:03:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:03:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:35328->10.242.163.157:8301: i/o timeout
2021/05/10 00:03:24 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:03:25 [ERR] memberlist: Push/Pull with dev-consul-server-2 failed: read tcp 10.242.61.70:35240->10.242.163.157:8301: i/o timeout
2021/05/10 00:03:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:35346->10.242.163.157:8301: i/o timeout
2021/05/10 00:03:27 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:03:28 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/05/10 00:03:28 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:03:28 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:03:30 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/10 00:03:30 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.163.138:32858
2021/05/10 00:03:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:51806->10.242.163.157:8302: i/o timeout
2021/05/10 00:03:33 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:03:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:51882->10.242.163.157:8302: i/o timeout
2021/05/10 00:03:43 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:03:46 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-xplxm 10.242.163.138:8301
2021/05/10 00:03:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:52014->10.242.163.157:8302: i/o timeout
2021/05/10 00:03:58 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:04:08 [ERR] yamux: keepalive failed: session shutdown
2021/05/10 00:04:09 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:04:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:45302->10.242.163.138:8301: i/o timeout
2021/05/10 00:04:10 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:04:12 [ERR] memberlist: failed to receive: read tcp 10.242.61.70:8301->10.242.163.157:45142: i/o timeout from=10.242.163.157:45142
2021/05/10 00:04:12 [ERR] memberlist: Failed to send error: write tcp 10.242.61.70:8301->10.242.163.157:45142: i/o timeout from=10.242.163.157:45142
2021/05/10 00:04:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:45312->10.242.163.138:8301: i/o timeout
2021/05/10 00:04:12 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:04:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:52130->10.242.163.157:8302: i/o timeout
2021/05/10 00:04:13 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:04:13 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/10 00:04:13 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.163.157
2021/05/10 00:04:13 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/05/10 00:04:14 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (0 peer confirmations)
2021/05/10 00:04:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:45326->10.242.163.138:8301: i/o timeout
2021/05/10 00:04:14 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:04:14 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:04:19 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.163.157
2021/05/10 00:04:19 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/05/10 00:04:26 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.163.157:8301
2021/05/10 00:04:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:52258->10.242.163.157:8302: i/o timeout
2021/05/10 00:04:28 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:04:35 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:04:35 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:04:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:35910->10.242.163.157:8301: i/o timeout
2021/05/10 00:04:36 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:04:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:35934->10.242.163.157:8301: i/o timeout
2021/05/10 00:04:38 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:04:41 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/10 00:04:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:36004->10.242.163.157:8301: i/o timeout
2021/05/10 00:04:46 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:04:48 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/10 00:04:49 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/10 00:04:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:36070->10.242.163.157:8301: i/o timeout
2021/05/10 00:04:53 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:04:56 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (0 peer confirmations)
2021/05/10 00:04:56 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:04:56 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:04:56 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:04:56 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:04:57 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-2wbhv 10.242.168.100
2021/05/10 00:04:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:36142->10.242.163.157:8301: i/o timeout
2021/05/10 00:04:57 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:04:59 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/10 00:05:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:36232->10.242.163.157:8301: i/o timeout
2021/05/10 00:05:03 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:05:04 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:05:04 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:05:05 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.163.157:8301
2021/05/10 00:05:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:36248->10.242.163.157:8301: i/o timeout
2021/05/10 00:05:06 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:05:09 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:05:09 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:05:09 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/05/10 00:05:09 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/05/10 00:05:12 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/10 00:05:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:36364->10.242.163.157:8301: i/o timeout
2021/05/10 00:05:15 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:05:18 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:05:18 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:05:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:52848->10.242.163.157:8302: i/o timeout
2021/05/10 00:05:23 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:05:24 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:05:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:46058->10.242.163.138:8301: i/o timeout
2021/05/10 00:05:25 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:05:28 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:05:28 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:05:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:46200->10.242.163.138:8301: i/o timeout
2021/05/10 00:05:42 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:05:45 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:05:46 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-xplxm but other probes failed, network may be misconfigured
2021/05/10 00:05:47 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:05:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.61.70:46300->10.242.163.138:8301: i/o timeout
2021/05/10 00:05:52 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:05:53 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (2 peer confirmations)
2021/05/10 00:05:53 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:05:56 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:06:10 [INFO] serf: EventMemberLeave: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:06:13 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:06:16 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/05/10 00:06:16 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:06:16 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:06:51 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/10 00:06:51 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.163.157
2021/05/10 00:06:51 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/05/10 00:06:51 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:06:59 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.163.157:8302
2021/05/10 00:06:59 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.183
2021/05/10 00:06:59 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.183:8300) (DC: dc1)
2021/05/10 00:06:59 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.163.157:8302 Theirs: 10.242.163.183:8302
2021/05/10 00:06:59 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.163.157:8302 and 10.242.163.183:8302 are claiming
2021/05/10 00:07:00 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.163.157:8302 Theirs: 10.242.163.183:8302
2021/05/10 00:07:00 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.163.157:8302 and 10.242.163.183:8302 are claiming
2021/05/10 00:07:00 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.163.157:8302 Theirs: 10.242.163.183:8302
2021/05/10 00:07:00 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.163.157:8302 and 10.242.163.183:8302 are claiming
2021/05/10 00:07:00 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.163.157:8302 Theirs: 10.242.163.183:8302
2021/05/10 00:07:00 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.163.157:8302 and 10.242.163.183:8302 are claiming
2021/05/10 00:07:00 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.163.157:8302 Theirs: 10.242.163.183:8302
2021/05/10 00:07:00 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.163.157:8302 and 10.242.163.183:8302 are claiming
2021/05/10 00:08:12 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.163.183
2021/05/10 00:08:12 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"