By type
+ CONSUL_BIND=
+ [ -n ]
+ CONSUL_CLIENT=
+ [ -n ]
+ CONSUL_DATA_DIR=/consul/data
+ CONSUL_CONFIG_DIR=/consul/config
+ [ -n ]
+ [ a = - ]
+ [ agent = agent ]
+ shift
+ set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
+ [ consul = consul ]
+ stat -c %u /consul/data
+ id -u consul
+ [ 100 != 100 ]
+ stat -c %u /consul/config
+ id -u consul
+ [ 100 != 100 ]
+ [ ! -z ]
+ exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
bootstrap_expect > 0: expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
Version: 'v1.0.6'
Node ID: 'c466abc0-d439-190d-61b3-a6edbb2267f8'
Node name: 'dev-consul-server-1'
Datacenter: 'dc1' (Segment: '')
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.242.225.64 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2021/09/27 11:32:33 [INFO] raft: Initial configuration (index=0): []
2021/09/27 11:32:33 [INFO] raft: Node at 10.242.225.64:8300 [Follower] entering Follower state (Leader: "")
2021/09/27 11:32:33 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.225.64
2021/09/27 11:32:33 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.225.64
2021/09/27 11:32:33 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/09/27 11:32:33 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2021/09/27 11:32:33 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.225.64:8300) (DC: dc1)
2021/09/27 11:32:33 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2021/09/27 11:32:33 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
2021/09/27 11:32:33 [INFO] agent: started state syncer
2021/09/27 11:32:33 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer
2021/09/27 11:32:33 [INFO] agent: Joining LAN cluster...
2021/09/27 11:32:33 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/09/27 11:32:33 [WARN] memberlist: Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
2021/09/27 11:32:33 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
2021/09/27 11:32:33 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/09/27 11:32:33 [INFO] agent: (LAN) joined: 0 Err: 3 error(s) occurred:
* Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/09/27 11:32:33 [WARN] agent: Join LAN failed: , retrying in 30s
==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=e0abfef7-58bd-ecae-4195-9547530c465c&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2021/09/27 11:32:39 [WARN] raft: no known peers, aborting election
2021/09/27 11:32:40 [ERR] agent: failed to sync remote state: No cluster leader
2021/09/27 11:32:46 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.28.11
2021/09/27 11:32:46 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:32:46 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 11:32:46 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.28.11
2021/09/27 11:32:46 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/09/27 11:32:56 [ERR] agent: Coordinate update error: No cluster leader
2021/09/27 11:33:03 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/09/27 11:33:03 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.59.64
2021/09/27 11:33:03 [INFO] agent: (LAN) joined: 3 Err:
2021/09/27 11:33:03 [INFO] agent: Join LAN completed. Synced with 3 initial agents
2021/09/27 11:33:03 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.59.64:8300) (DC: dc1)
2021/09/27 11:33:03 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.242.225.64:8300,10.242.28.11:8300,10.242.59.64:8300
2021/09/27 11:33:03 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.59.64
2021/09/27 11:33:03 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/09/27 11:33:03 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/09/27 11:33:03 [INFO] raft: Node at 10.242.225.64:8300 [Candidate] entering Candidate state in term 2
2021/09/27 11:33:03 [INFO] raft: Election won. Tally: 2
2021/09/27 11:33:03 [INFO] raft: Node at 10.242.225.64:8300 [Leader] entering Leader state
2021/09/27 11:33:03 [INFO] raft: Added peer 8262321d-b29f-83df-f962-bd890d88abc1, starting replication
2021/09/27 11:33:03 [INFO] raft: Added peer 0044e147-4efb-f870-c8eb-6a34a13674cf, starting replication
2021/09/27 11:33:03 [INFO] consul: cluster leadership acquired
2021/09/27 11:33:03 [INFO] consul: New leader elected: dev-consul-server-1
2021/09/27 11:33:03 [WARN] raft: AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300} rejected, sending older logs (next: 1)
2021/09/27 11:33:03 [WARN] raft: AppendEntries to {Voter 0044e147-4efb-f870-c8eb-6a34a13674cf 10.242.59.64:8300} rejected, sending older logs (next: 1)
2021/09/27 11:33:03 [INFO] raft: pipelining replication to peer {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}
2021/09/27 11:33:03 [INFO] raft: pipelining replication to peer {Voter 0044e147-4efb-f870-c8eb-6a34a13674cf 10.242.59.64:8300}
2021/09/27 11:33:03 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/09/27 11:33:03 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/09/27 11:33:03 [INFO] agent: Synced node info
2021/09/27 11:33:03 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 11:33:03 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/09/27 11:43:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:40296->10.242.28.28:8301: i/o timeout
2021/09/27 11:43:46 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:43:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:40304->10.242.28.28:8301: i/o timeout
2021/09/27 11:43:50 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:43:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:40308->10.242.28.28:8301: i/o timeout
2021/09/27 11:43:53 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:43:54 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (2 peer confirmations)
2021/09/27 11:43:54 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:43:54 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 11:43:55 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/09/27 11:43:57 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/09/27 11:43:58 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/09/27 11:44:02 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:44:02 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 11:44:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:40366->10.242.28.28:8301: i/o timeout
2021/09/27 11:44:26 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:44:26 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500180319s
2021/09/27 11:44:26 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 11:44:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 11:44:27 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:44:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:40368->10.242.28.28:8301: i/o timeout
2021/09/27 11:44:28 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:44:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:40376->10.242.28.28:8301: i/o timeout
2021/09/27 11:44:31 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:44:32 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 11:44:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43052->10.242.28.11:8301: i/o timeout
2021/09/27 11:44:33 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:44:33 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500317349s
2021/09/27 11:44:33 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:44:33 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 11:44:34 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:44:34 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 11:45:26 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 11:45:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43140->10.242.28.11:8301: i/o timeout
2021/09/27 11:45:27 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:46:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:40558->10.242.28.28:8301: i/o timeout
2021/09/27 11:46:28 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:46:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:40572->10.242.28.28:8301: i/o timeout
2021/09/27 11:46:34 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:46:34 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 11:46:35 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:46:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43248->10.242.28.11:8301: i/o timeout
2021/09/27 11:46:37 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:46:37 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:46:39 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:46:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43256->10.242.28.11:8301: i/o timeout
2021/09/27 11:46:40 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:46:40 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500331616s
2021/09/27 11:46:41 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.28.11
2021/09/27 11:46:41 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 11:46:41 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/09/27 11:46:41 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:46:42 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.28.11
2021/09/27 11:46:42 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 11:46:42 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/09/27 11:46:43 [INFO] raft: aborting pipeline replication to peer {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}
2021/09/27 11:46:43 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:46:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43266->10.242.28.11:8301: i/o timeout
2021/09/27 11:46:44 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:46:44 [INFO] raft: pipelining replication to peer {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}
2021/09/27 11:47:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:40690->10.242.28.28:8301: i/o timeout
2021/09/27 11:47:52 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:47:54 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 11:47:54 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 11:47:54 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/09/27 11:47:55 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/09/27 11:47:55 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500294971s
2021/09/27 11:47:55 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:47:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43370->10.242.28.11:8301: i/o timeout
2021/09/27 11:47:57 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:47:57 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:47:58 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.950039481s
2021/09/27 11:47:58 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (2 peer confirmations)
2021/09/27 11:47:58 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:47:58 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 11:47:59 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:47:59 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 11:48:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:40728->10.242.28.28:8301: i/o timeout
2021/09/27 11:48:12 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:48:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:40736->10.242.28.28:8301: i/o timeout
2021/09/27 11:48:16 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:48:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43446->10.242.28.11:8301: i/o timeout
2021/09/27 11:48:29 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:48:30 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 11:48:30 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 11:48:31 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:48:31 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500302145s
2021/09/27 11:48:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:40776->10.242.28.28:8301: i/o timeout
2021/09/27 11:48:32 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:48:33 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:48:34 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.953656624s
2021/09/27 11:48:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:40784->10.242.28.28:8301: i/o timeout
2021/09/27 11:48:35 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:48:35 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:48:36 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 7.443729456s
2021/09/27 11:48:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43460->10.242.28.11:8301: i/o timeout
2021/09/27 11:48:37 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:48:37 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (2 peer confirmations)
2021/09/27 11:48:37 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:48:37 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 11:48:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43488->10.242.28.11:8301: i/o timeout
2021/09/27 11:48:44 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:48:44 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.50033507s
2021/09/27 11:48:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43492->10.242.28.11:8301: i/o timeout
2021/09/27 11:48:46 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:48:54 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:48:54 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 11:49:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:40870->10.242.28.28:8301: i/o timeout
2021/09/27 11:49:17 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:49:21 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (2 peer confirmations)
2021/09/27 11:49:21 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:49:21 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 11:49:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:40880->10.242.28.28:8301: i/o timeout
2021/09/27 11:49:21 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:49:22 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:49:22 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 11:49:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:40916->10.242.28.28:8301: i/o timeout
2021/09/27 11:49:44 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:50:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:40946->10.242.28.28:8301: i/o timeout
2021/09/27 11:50:06 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:50:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:40952->10.242.28.28:8301: i/o timeout
2021/09/27 11:50:09 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:50:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:40960->10.242.28.28:8301: i/o timeout
2021/09/27 11:50:11 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:50:12 [WARN] memberlist: Refuting a suspect message (from: dev-consul-68b98449d-z4w69)
2021/09/27 11:50:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:40970->10.242.28.28:8301: i/o timeout
2021/09/27 11:50:16 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:50:36 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 11:50:36 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 11:50:36 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/09/27 11:50:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41002->10.242.28.28:8301: i/o timeout
2021/09/27 11:50:37 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:50:37 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (2 peer confirmations)
2021/09/27 11:50:37 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:50:37 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 11:50:38 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 11:50:38 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 11:50:39 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:50:40 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/09/27 11:50:40 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500277147s
2021/09/27 11:50:41 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:50:42 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.987208163s
2021/09/27 11:50:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43686->10.242.28.11:8301: i/o timeout
2021/09/27 11:50:43 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:50:43 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:50:44 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/09/27 11:50:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43692->10.242.28.11:8301: i/o timeout
2021/09/27 11:50:45 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:50:45 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:50:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43696->10.242.28.11:8301: i/o timeout
2021/09/27 11:50:48 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:50:48 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 11:50:48 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 11:50:53 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:50:53 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 11:51:14 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 11:51:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43736->10.242.28.11:8301: i/o timeout
2021/09/27 11:51:15 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:51:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41094->10.242.28.28:8301: i/o timeout
2021/09/27 11:51:30 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:51:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41100->10.242.28.28:8301: i/o timeout
2021/09/27 11:51:33 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:51:34 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:51:34 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 11:51:48 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:51:48 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 11:52:01 [WARN] memberlist: Was able to connect to dev-consul-68b98449d-z4w69 but other probes failed, network may be misconfigured
2021/09/27 11:52:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41184->10.242.28.28:8301: i/o timeout
2021/09/27 11:52:25 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:52:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41192->10.242.28.28:8301: i/o timeout
2021/09/27 11:52:29 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:53:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41234->10.242.28.28:8301: i/o timeout
2021/09/27 11:53:02 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:53:11 [WARN] memberlist: Was able to connect to dev-consul-68b98449d-z4w69 but other probes failed, network may be misconfigured
2021/09/27 11:53:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41274->10.242.28.28:8301: i/o timeout
2021/09/27 11:53:27 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:53:27 [ERR] yamux: keepalive failed: session shutdown
2021/09/27 11:53:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41294->10.242.28.28:8301: i/o timeout
2021/09/27 11:53:40 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:54:30 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 11:54:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:44122->10.242.28.11:8301: i/o timeout
2021/09/27 11:54:32 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:54:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41458->10.242.28.28:8301: i/o timeout
2021/09/27 11:54:34 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:54:34 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 11:54:35 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:54:36 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2021/09/27 11:54:36 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.28.11
2021/09/27 11:54:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:44132->10.242.28.11:8301: i/o timeout
2021/09/27 11:54:36 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:54:36 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 11:54:36 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/09/27 11:54:36 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.50021119s
2021/09/27 11:54:37 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:54:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41462->10.242.28.28:8301: i/o timeout
2021/09/27 11:54:38 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:54:39 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.996629005s
2021/09/27 11:54:39 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.28.11
2021/09/27 11:54:39 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 11:54:39 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/09/27 11:54:39 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:54:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41470->10.242.28.28:8301: i/o timeout
2021/09/27 11:54:40 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:54:41 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 7.444460502s
2021/09/27 11:54:41 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:54:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:44144->10.242.28.11:8301: i/o timeout
2021/09/27 11:54:42 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:54:43 [WARN] memberlist: Was able to connect to dev-consul-server-0.dc1 but other probes failed, network may be misconfigured
2021/09/27 11:54:43 [INFO] raft: aborting pipeline replication to peer {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}
2021/09/27 11:54:43 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:54:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41478->10.242.28.28:8301: i/o timeout
2021/09/27 11:54:44 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:54:45 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (1 peer confirmations)
2021/09/27 11:54:45 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:54:45 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 11:54:45 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500375557s
2021/09/27 11:54:45 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:54:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:44158->10.242.28.11:8301: i/o timeout
2021/09/27 11:54:46 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:54:47 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:54:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:40620->10.242.28.11:8302: i/o timeout
2021/09/27 11:54:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:44162->10.242.28.11:8301: i/o timeout
2021/09/27 11:54:48 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:54:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:44172->10.242.28.11:8301: i/o timeout
2021/09/27 11:54:50 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:54:52 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.50213998s
2021/09/27 11:54:52 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 11:54:52 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 11:54:53 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:56230->10.242.28.11:8300: i/o timeout
2021/09/27 11:54:53 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:54:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:44190->10.242.28.11:8301: i/o timeout
2021/09/27 11:54:54 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:54:56 [INFO] raft: pipelining replication to peer {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}
2021/09/27 11:54:58 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:54:58 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 11:55:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41550->10.242.28.28:8301: i/o timeout
2021/09/27 11:55:02 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:55:02 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 11:55:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41564->10.242.28.28:8301: i/o timeout
2021/09/27 11:55:05 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:55:06 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 11:55:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:44242->10.242.28.11:8301: i/o timeout
2021/09/27 11:55:07 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:55:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41646->10.242.28.28:8301: i/o timeout
2021/09/27 11:55:19 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:55:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41658->10.242.28.28:8301: i/o timeout
2021/09/27 11:55:22 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:55:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41694->10.242.28.28:8301: i/o timeout
2021/09/27 11:55:26 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:55:26 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (2 peer confirmations)
2021/09/27 11:55:26 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:55:26 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 11:55:26 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:55:26 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 11:55:28 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 11:55:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41794->10.242.28.28:8301: i/o timeout
2021/09/27 11:55:40 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:55:40 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 11:56:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41862->10.242.28.28:8301: i/o timeout
2021/09/27 11:56:15 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:56:34 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 11:56:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41952->10.242.28.28:8301: i/o timeout
2021/09/27 11:56:35 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:56:36 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/09/27 11:56:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41958->10.242.28.28:8301: i/o timeout
2021/09/27 11:56:39 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:56:39 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (2 peer confirmations)
2021/09/27 11:56:39 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:56:39 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 11:56:40 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:56:40 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 11:56:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41972->10.242.28.28:8301: i/o timeout
2021/09/27 11:56:44 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:56:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41982->10.242.28.28:8301: i/o timeout
2021/09/27 11:56:47 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:56:47 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:56:47 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 11:56:48 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 11:56:49 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500451521s
2021/09/27 11:56:49 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:56:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:44666->10.242.28.11:8301: i/o timeout
2021/09/27 11:56:50 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:56:51 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:56:52 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.969737946s
2021/09/27 11:56:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:44688->10.242.28.11:8301: i/o timeout
2021/09/27 11:56:53 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:56:53 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:56:54 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 7.453004317s
2021/09/27 11:56:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:44696->10.242.28.11:8301: i/o timeout
2021/09/27 11:56:55 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 11:56:55 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:56:57 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/09/27 11:56:57 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:56:57 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 11:57:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42070->10.242.28.28:8301: i/o timeout
2021/09/27 11:57:15 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:57:17 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (2 peer confirmations)
2021/09/27 11:57:17 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:57:17 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 11:57:18 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:57:18 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 11:57:19 [WARN] memberlist: Refuting a suspect message (from: dev-consul-68b98449d-z4w69)
2021/09/27 11:57:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42124->10.242.28.28:8301: i/o timeout
2021/09/27 11:57:42 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:57:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42132->10.242.28.28:8301: i/o timeout
2021/09/27 11:57:47 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:57:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42144->10.242.28.28:8301: i/o timeout
2021/09/27 11:57:53 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:57:56 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:57:56 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 11:57:56 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 11:57:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42152->10.242.28.28:8301: i/o timeout
2021/09/27 11:57:57 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:57:57 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500345846s
2021/09/27 11:57:57 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 11:58:00 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:58:00 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 11:58:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42198->10.242.28.28:8301: i/o timeout
2021/09/27 11:58:14 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:58:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42202->10.242.28.28:8301: i/o timeout
2021/09/27 11:58:17 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 11:58:17 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (2 peer confirmations)
2021/09/27 11:58:17 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:58:17 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 11:58:25 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 11:58:25 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 11:59:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42374->10.242.28.28:8301: i/o timeout
2021/09/27 11:59:51 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:00:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42396->10.242.28.28:8301: i/o timeout
2021/09/27 12:00:00 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:00:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42404->10.242.28.28:8301: i/o timeout
2021/09/27 12:00:04 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:00:06 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:00:06 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:00:07 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500312416s
2021/09/27 12:00:07 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:00:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42414->10.242.28.28:8301: i/o timeout
2021/09/27 12:00:08 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:00:08 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (2 peer confirmations)
2021/09/27 12:00:08 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:00:08 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 12:00:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42422->10.242.28.28:8301: i/o timeout
2021/09/27 12:00:11 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:00:15 [WARN] memberlist: Refuting a suspect message (from: dev-consul-68b98449d-z4w69)
2021/09/27 12:00:15 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:00:15 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 12:00:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42452->10.242.28.28:8301: i/o timeout
2021/09/27 12:00:25 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:00:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42462->10.242.28.28:8301: i/o timeout
2021/09/27 12:00:28 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:00:28 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:00:28 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 12:00:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42470->10.242.28.28:8301: i/o timeout
2021/09/27 12:00:31 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:00:31 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:00:31 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 12:00:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42496->10.242.28.28:8301: i/o timeout
2021/09/27 12:00:44 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:00:44 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:00:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42500->10.242.28.28:8301: i/o timeout
2021/09/27 12:00:47 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:00:48 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (2 peer confirmations)
2021/09/27 12:00:48 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:00:48 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 12:00:50 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:00:51 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:00:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45190->10.242.28.11:8301: i/o timeout
2021/09/27 12:00:55 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:00:56 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500397148s
2021/09/27 12:00:56 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:00:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45194->10.242.28.11:8301: i/o timeout
2021/09/27 12:00:57 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:00:57 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:00:58 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.925921581s
2021/09/27 12:00:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45206->10.242.28.11:8301: i/o timeout
2021/09/27 12:00:59 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:00:59 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:01:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45208->10.242.28.11:8301: i/o timeout
2021/09/27 12:01:01 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:01:02 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500243532s
2021/09/27 12:01:02 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:01:03 [INFO] serf: attempting reconnect to dev-consul-68b98449d-z4w69 10.242.28.28:8301
2021/09/27 12:01:05 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:01:05 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 12:01:19 [WARN] memberlist: Was able to connect to dev-consul-68b98449d-z4w69 but other probes failed, network may be misconfigured
2021/09/27 12:01:20 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:01:20 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:01:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45250->10.242.28.11:8301: i/o timeout
2021/09/27 12:01:21 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:01:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45266->10.242.28.11:8301: i/o timeout
2021/09/27 12:01:26 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:01:28 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500276618s
2021/09/27 12:01:28 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:01:30 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:01:30 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:01:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45278->10.242.28.11:8301: i/o timeout
2021/09/27 12:01:31 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:01:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42610->10.242.28.28:8301: i/o timeout
2021/09/27 12:01:34 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:01:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42620->10.242.28.28:8301: i/o timeout
2021/09/27 12:01:38 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:01:38 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (2 peer confirmations)
2021/09/27 12:01:38 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:01:38 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 12:01:38 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:01:39 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:01:41 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:01:42 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500272186s
2021/09/27 12:01:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45302->10.242.28.11:8301: i/o timeout
2021/09/27 12:01:43 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:01:43 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:01:51 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/09/27 12:01:52 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:01:53 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:01:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45328->10.242.28.11:8301: i/o timeout
2021/09/27 12:01:54 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:01:54 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500322261s
2021/09/27 12:01:55 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:01:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45330->10.242.28.11:8301: i/o timeout
2021/09/27 12:01:56 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.936120618s
2021/09/27 12:01:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45334->10.242.28.11:8301: i/o timeout
2021/09/27 12:01:57 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:01:57 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:01:58 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 7.403028178s
2021/09/27 12:01:59 [INFO] raft: aborting pipeline replication to peer {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}
2021/09/27 12:01:59 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:02:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45348->10.242.28.11:8301: i/o timeout
2021/09/27 12:02:01 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:02:01 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:02:02 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500399188s
2021/09/27 12:02:02 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:54082->10.242.28.11:8300: i/o timeout
2021/09/27 12:02:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41810->10.242.28.11:8302: i/o timeout
2021/09/27 12:02:03 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:02:03 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:02:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45360->10.242.28.11:8301: i/o timeout
2021/09/27 12:02:04 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:02:04 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.28.11
2021/09/27 12:02:04 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 12:02:04 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/09/27 12:02:04 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.977186824s
2021/09/27 12:02:04 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.28.11:8301
2021/09/27 12:02:05 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:02:07 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 7.476198849s
2021/09/27 12:02:07 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:02:09 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:57420->10.242.28.11:8300: i/o timeout
2021/09/27 12:02:09 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:02:09 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:02:11 [ERR] yamux: keepalive failed: i/o deadline reached
2021/09/27 12:02:11 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/09/27 12:02:12 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:02:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:02:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:41840->10.242.28.11:8302: i/o timeout
2021/09/27 12:02:13 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:02:13 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:57432->10.242.28.11:8300: i/o timeout
2021/09/27 12:02:13 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:02:14 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:02:15 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:02:17 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:02:17 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 12:02:17 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:02:18 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.28.11
2021/09/27 12:02:18 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 12:02:18 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/09/27 12:02:19 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:57452->10.242.28.11:8300: i/o timeout
2021/09/27 12:02:19 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:02:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42730->10.242.28.28:8301: i/o timeout
2021/09/27 12:02:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45406->10.242.28.11:8301: i/o timeout
2021/09/27 12:02:21 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:02:21 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.50027987s
2021/09/27 12:02:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45412->10.242.28.11:8301: i/o timeout
2021/09/27 12:02:24 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:02:24 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:02:25 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:02:26 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/09/27 12:02:26 [INFO] raft: pipelining replication to peer {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}
2021/09/27 12:03:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42892->10.242.28.28:8301: i/o timeout
2021/09/27 12:03:43 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:03:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42900->10.242.28.28:8301: i/o timeout
2021/09/27 12:03:46 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:03:46 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:03:46 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 12:03:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45584->10.242.28.11:8301: i/o timeout
2021/09/27 12:03:52 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:04:12 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:04:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:04:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45624->10.242.28.11:8301: i/o timeout
2021/09/27 12:04:13 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:04:13 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:04:14 [INFO] serf: attempting reconnect to dev-consul-68b98449d-z4w69 10.242.28.28:8301
2021/09/27 12:04:18 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/09/27 12:04:21 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:04:21 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 12:04:22 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:04:22 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:04:24 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/09/27 12:04:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42990->10.242.28.28:8301: i/o timeout
2021/09/27 12:04:26 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:04:27 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:04:27 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 12:04:28 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:04:28 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 12:04:52 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:04:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43036->10.242.28.28:8301: i/o timeout
2021/09/27 12:04:53 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:04:53 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:04:55 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500265292s
2021/09/27 12:04:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43042->10.242.28.28:8301: i/o timeout
2021/09/27 12:04:56 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:04:57 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (2 peer confirmations)
2021/09/27 12:04:57 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:04:57 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 12:04:59 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:04:59 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 12:05:00 [WARN] memberlist: Refuting a suspect message (from: dev-consul-68b98449d-z4w69)
2021/09/27 12:05:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43078->10.242.28.28:8301: i/o timeout
2021/09/27 12:05:14 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:05:14 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:05:14 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:05:15 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500261359s
2021/09/27 12:05:15 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:05:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43082->10.242.28.28:8301: i/o timeout
2021/09/27 12:05:17 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:05:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43092->10.242.28.28:8301: i/o timeout
2021/09/27 12:05:24 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:05:30 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:05:30 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:05:32 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:05:32 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:05:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43120->10.242.28.28:8301: i/o timeout
2021/09/27 12:05:33 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:05:35 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/09/27 12:05:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43128->10.242.28.28:8301: i/o timeout
2021/09/27 12:05:37 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:05:38 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (2 peer confirmations)
2021/09/27 12:05:38 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:05:38 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 12:05:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43132->10.242.28.28:8301: i/o timeout
2021/09/27 12:05:40 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:05:40 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:05:40 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:05:41 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500385926s
2021/09/27 12:05:41 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:05:42 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:05:42 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 12:05:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43148->10.242.28.28:8301: i/o timeout
2021/09/27 12:05:45 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:05:46 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:05:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43152->10.242.28.28:8301: i/o timeout
2021/09/27 12:05:48 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:05:48 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:05:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45834->10.242.28.11:8301: i/o timeout
2021/09/27 12:05:51 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:05:52 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500356753s
2021/09/27 12:05:52 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:05:52 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:05:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:42296->10.242.28.11:8302: i/o timeout
2021/09/27 12:05:53 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:05:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43168->10.242.28.28:8301: i/o timeout
2021/09/27 12:05:53 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:05:53 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:05:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43174->10.242.28.28:8301: i/o timeout
2021/09/27 12:05:55 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:05:55 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:05:56 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (1 peer confirmations)
2021/09/27 12:05:56 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:05:56 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 12:05:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45852->10.242.28.11:8301: i/o timeout
2021/09/27 12:06:06 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:06:07 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500242207s
2021/09/27 12:06:07 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:06:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45878->10.242.28.11:8301: i/o timeout
2021/09/27 12:06:08 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:06:09 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:06:12 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500319488s
2021/09/27 12:06:12 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:06:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45894->10.242.28.11:8301: i/o timeout
2021/09/27 12:06:13 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:06:13 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:06:16 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:06:17 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:06:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45906->10.242.28.11:8301: i/o timeout
2021/09/27 12:06:18 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:06:18 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500348325s
2021/09/27 12:06:19 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:06:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45914->10.242.28.11:8301: i/o timeout
2021/09/27 12:06:20 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:06:20 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.938866912s
2021/09/27 12:06:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45934->10.242.28.11:8301: i/o timeout
2021/09/27 12:06:26 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:06:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:45946->10.242.28.11:8301: i/o timeout
2021/09/27 12:06:32 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:06:33 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:06:34 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:06:34 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 12:06:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43286->10.242.28.28:8301: i/o timeout
2021/09/27 12:06:36 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:06:36 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:06:36 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:06:40 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:06:40 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 12:06:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43298->10.242.28.28:8301: i/o timeout
2021/09/27 12:06:41 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:06:42 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:06:42 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 12:06:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43330->10.242.28.28:8301: i/o timeout
2021/09/27 12:06:56 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:06:56 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:06:57 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:07:08 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/09/27 12:07:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43362->10.242.28.28:8301: i/o timeout
2021/09/27 12:07:11 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:07:12 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:07:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:07:13 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500276937s
2021/09/27 12:07:13 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:07:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:46042->10.242.28.11:8301: i/o timeout
2021/09/27 12:07:14 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:07:16 [WARN] memberlist: Was able to connect to dev-consul-68b98449d-z4w69 but other probes failed, network may be misconfigured
2021/09/27 12:07:18 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:07:18 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:07:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43386->10.242.28.28:8301: i/o timeout
2021/09/27 12:07:19 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:07:19 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:07:20 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:07:20 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 12:07:26 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:07:27 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:07:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:46084->10.242.28.11:8301: i/o timeout
2021/09/27 12:07:29 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:07:29 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:07:31 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/09/27 12:07:32 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:07:32 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 12:07:40 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:07:42 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:07:42 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:07:43 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500315279s
2021/09/27 12:07:43 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:07:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43552->10.242.28.28:8301: i/o timeout
2021/09/27 12:07:45 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:07:45 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:07:46 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.996946027s
2021/09/27 12:07:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:46248->10.242.28.11:8301: i/o timeout
2021/09/27 12:07:47 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:07:47 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:07:48 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 7.425359303s
2021/09/27 12:07:48 [DEBUG] raft-net: 10.242.225.64:8300 accepted connection from: 10.242.28.11:53166
2021/09/27 12:07:48 [WARN] raft: Rejecting vote request from 10.242.28.11:8300 since we have a leader: 10.242.225.64:8300
2021/09/27 12:07:48 [ERR] raft: peer {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300} has newer term, stopping replication
2021/09/27 12:07:48 [INFO] raft: aborting pipeline replication to peer {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}
2021/09/27 12:07:48 [INFO] raft: Node at 10.242.225.64:8300 [Follower] entering Follower state (Leader: "")
2021/09/27 12:07:48 [INFO] consul: cluster leadership lost
2021/09/27 12:07:48 [INFO] raft: aborting pipeline replication to peer {Voter 0044e147-4efb-f870-c8eb-6a34a13674cf 10.242.59.64:8300}
2021/09/27 12:07:49 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:07:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43652->10.242.28.28:8301: i/o timeout
2021/09/27 12:07:50 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:07:53 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:07:56 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/09/27 12:07:56 [INFO] raft: Node at 10.242.225.64:8300 [Candidate] entering Candidate state in term 3
2021/09/27 12:07:56 [INFO] raft: Election won. Tally: 2
2021/09/27 12:07:56 [INFO] raft: Node at 10.242.225.64:8300 [Leader] entering Leader state
2021/09/27 12:07:56 [INFO] raft: Added peer 8262321d-b29f-83df-f962-bd890d88abc1, starting replication
2021/09/27 12:07:56 [INFO] raft: Added peer 0044e147-4efb-f870-c8eb-6a34a13674cf, starting replication
2021/09/27 12:07:56 [INFO] consul: cluster leadership acquired
2021/09/27 12:07:56 [INFO] consul: New leader elected: dev-consul-server-1
2021/09/27 12:07:56 [INFO] raft: pipelining replication to peer {Voter 0044e147-4efb-f870-c8eb-6a34a13674cf 10.242.59.64:8300}
2021/09/27 12:07:57 [WARN] raft: AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300} rejected, sending older logs (next: 1055)
2021/09/27 12:07:57 [INFO] raft: pipelining replication to peer {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}
2021/09/27 12:07:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:46502->10.242.28.11:8301: i/o timeout
2021/09/27 12:07:59 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:07:59 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:07:59 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:08:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:03 [WARN] memberlist: Was able to connect to dev-consul-68b98449d-z4w69 but other probes failed, network may be misconfigured
2021/09/27 12:08:03 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:08:03 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:08:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43936->10.242.28.28:8301: i/o timeout
2021/09/27 12:08:05 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:08:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:06 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.50023879s
2021/09/27 12:08:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:46652->10.242.28.11:8301: i/o timeout
2021/09/27 12:08:07 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:08:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43102->10.242.28.11:8302: i/o timeout
2021/09/27 12:08:08 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:08:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:46676->10.242.28.11:8301: i/o timeout
2021/09/27 12:08:09 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:08:09 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.984773919s
2021/09/27 12:08:09 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:08:09 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:08:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:44054->10.242.28.28:8301: i/o timeout
2021/09/27 12:08:11 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:08:11 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 7.433496211s
2021/09/27 12:08:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:46774->10.242.28.11:8301: i/o timeout
2021/09/27 12:08:13 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:08:14 [INFO] raft: aborting pipeline replication to peer {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}
2021/09/27 12:08:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:44142->10.242.28.28:8301: i/o timeout
2021/09/27 12:08:15 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:08:15 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:57464->10.242.28.11:8300: i/o timeout
2021/09/27 12:08:15 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations)
2021/09/27 12:08:15 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.28.11
2021/09/27 12:08:15 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 12:08:15 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/09/27 12:08:15 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:08:16 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (1 peer confirmations)
2021/09/27 12:08:16 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:08:16 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 12:08:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:44192->10.242.28.28:8301: i/o timeout
2021/09/27 12:08:17 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:08:17 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500375461s
2021/09/27 12:08:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43312->10.242.28.11:8302: i/o timeout
2021/09/27 12:08:18 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:08:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:19 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.922832079s
2021/09/27 12:08:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:21 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 7.383587572s
2021/09/27 12:08:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:23 [INFO] serf: attempting reconnect to dev-consul-68b98449d-z4w69 10.242.28.28:8301
2021/09/27 12:08:24 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:58894->10.242.28.11:8300: i/o timeout
2021/09/27 12:08:24 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:08:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:25 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:58922->10.242.28.11:8300: i/o timeout
2021/09/27 12:08:26 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:08:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:43422->10.242.28.11:8302: i/o timeout
2021/09/27 12:08:28 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:08:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:31 [ERR] yamux: keepalive failed: i/o deadline reached
2021/09/27 12:08:31 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/09/27 12:08:33 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:08:34 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:59098->10.242.28.11:8300: i/o timeout
2021/09/27 12:08:34 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:08:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:36 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:59148->10.242.28.11:8300: i/o timeout
2021/09/27 12:08:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:36 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/09/27 12:08:36 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.28.11
2021/09/27 12:08:36 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/09/27 12:08:37 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:08:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:42 [ERR] yamux: keepalive failed: i/o deadline reached
2021/09/27 12:08:42 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.28.28:33104
2021/09/27 12:08:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:44 [ERR] yamux: keepalive failed: i/o deadline reached
2021/09/27 12:08:44 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.28.11:50836
2021/09/27 12:08:44 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:59318->10.242.28.11:8300: i/o timeout
2021/09/27 12:08:44 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:08:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:47 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:59386->10.242.28.11:8300: i/o timeout
2021/09/27 12:08:48 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:08:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:54 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:59518->10.242.28.11:8300: i/o timeout
2021/09/27 12:08:54 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:08:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:58 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:59598->10.242.28.11:8300: i/o timeout
2021/09/27 12:08:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:08:59 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:09:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:03 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.28.11:8302
2021/09/27 12:09:03 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.28.11:8301
2021/09/27 12:09:04 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:59732->10.242.28.11:8300: i/o timeout
2021/09/27 12:09:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:04 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:09:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:09 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:59796->10.242.28.11:8300: i/o timeout
2021/09/27 12:09:10 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:09:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:12 [ERR] yamux: keepalive failed: i/o deadline reached
2021/09/27 12:09:12 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/09/27 12:09:14 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:59918->10.242.28.11:8300: i/o timeout
2021/09/27 12:09:15 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:09:15 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:09:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:20 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:60032->10.242.28.11:8300: i/o timeout
2021/09/27 12:09:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:21 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:09:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:25 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:60140->10.242.28.11:8300: i/o timeout
2021/09/27 12:09:25 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:09:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:31 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:60252->10.242.28.11:8300: i/o timeout
2021/09/27 12:09:32 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:09:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:35 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:60352->10.242.28.11:8300: i/o timeout
2021/09/27 12:09:36 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:09:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:42 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:60482->10.242.28.11:8300: i/o timeout
2021/09/27 12:09:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:43 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.28.11:8301
2021/09/27 12:09:44 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:09:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:46 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:60560->10.242.28.11:8300: i/o timeout
2021/09/27 12:09:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:47 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:09:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:54 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:60726->10.242.28.11:8300: i/o timeout
2021/09/27 12:09:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:54 [ERR] yamux: keepalive failed: i/o deadline reached
2021/09/27 12:09:54 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/09/27 12:09:57 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:60786->10.242.28.11:8300: i/o timeout
2021/09/27 12:09:57 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:09:57 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:09:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:09:59 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:10:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:07 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:60978->10.242.28.11:8300: i/o timeout
2021/09/27 12:10:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:09 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:32796->10.242.28.11:8300: i/o timeout
2021/09/27 12:10:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:13 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:10:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:15 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:10:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:23 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:33062->10.242.28.11:8300: i/o timeout
2021/09/27 12:10:23 [INFO] serf: attempting reconnect to dev-consul-68b98449d-z4w69 10.242.28.28:8301
2021/09/27 12:10:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:25 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:33064->10.242.28.11:8300: i/o timeout
2021/09/27 12:10:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:34 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:10:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:35 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:10:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:36 [ERR] yamux: keepalive failed: i/o deadline reached
2021/09/27 12:10:36 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/09/27 12:10:39 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:10:39 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:10:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:44 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:33438->10.242.28.11:8300: i/o timeout
2021/09/27 12:10:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:45 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:33452->10.242.28.11:8300: i/o timeout
2021/09/27 12:10:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:55 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:10:55 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:10:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:10:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:11:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:11:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:11:03 [INFO] serf: attempting reconnect to dev-consul-68b98449d-z4w69 10.242.28.28:8301
2021/09/27 12:11:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:11:05 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:33786->10.242.28.11:8300: i/o timeout
2021/09/27 12:11:05 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:33796->10.242.28.11:8300: i/o timeout
2021/09/27 12:11:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:11:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:11:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:11:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:11:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:11:15 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:11:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:11:17 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:11:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:11:18 [ERR] yamux: keepalive failed: i/o deadline reached
2021/09/27 12:11:18 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/09/27 12:11:21 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:11:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:11:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:11:25 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:34190->10.242.28.11:8300: i/o timeout
2021/09/27 12:11:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:11:27 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:34220->10.242.28.11:8300: i/o timeout
2021/09/27 12:11:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:11:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:11:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:11:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:11:36 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:11:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:11:38 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:11:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:11:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:11:41 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.28.11
2021/09/27 12:11:41 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:11:41 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 12:11:41 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/09/27 12:11:41 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 12:11:42 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.28.11
2021/09/27 12:11:42 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/09/27 12:11:42 [WARN] raft: Rejecting vote request from 10.242.28.11:8300 since we have a leader: 10.242.225.64:8300
2021/09/27 12:11:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:11:44 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/09/27 12:11:44 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:11:44 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 12:11:44 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:11:44 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 12:11:44 [WARN] memberlist: Refuting a suspect message (from: dev-consul-68b98449d-z4w69)
2021/09/27 12:11:44 [INFO] raft: pipelining replication to peer {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}
2021/09/27 12:12:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:48850->10.242.28.28:8301: i/o timeout
2021/09/27 12:12:15 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:12:17 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:12:17 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:12:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:51588->10.242.28.11:8301: i/o timeout
2021/09/27 12:12:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:12:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:48944->10.242.28.28:8301: i/o timeout
2021/09/27 12:12:19 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:12:19 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (2 peer confirmations)
2021/09/27 12:12:19 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:12:19 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 12:12:21 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:12:21 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:12:21 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/09/27 12:12:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:51658->10.242.28.11:8301: i/o timeout
2021/09/27 12:12:22 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:12:24 [ERR] yamux: keepalive failed: i/o deadline reached
2021/09/27 12:12:24 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.28.28:37830
2021/09/27 12:12:29 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:12:29 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 12:12:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:49210->10.242.28.28:8301: i/o timeout
2021/09/27 12:12:33 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:12:33 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:12:33 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:12:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:12:35 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500217924s
2021/09/27 12:12:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:51908->10.242.28.11:8301: i/o timeout
2021/09/27 12:12:36 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:12:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:12:37 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.935625928s
2021/09/27 12:12:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:12:39 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2021/09/27 12:12:39 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.28.11
2021/09/27 12:12:39 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 12:12:39 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/09/27 12:12:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:51992->10.242.28.11:8301: i/o timeout
2021/09/27 12:12:40 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:12:40 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:12:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:12:42 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500380074s
2021/09/27 12:12:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:12:43 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.28.11:8301
2021/09/27 12:12:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:49424->10.242.28.28:8301: i/o timeout
2021/09/27 12:12:44 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:12:44 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/09/27 12:12:44 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:12:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:12:46 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.28.11
2021/09/27 12:12:46 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 12:12:46 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/09/27 12:12:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:49458->10.242.28.28:8301: i/o timeout
2021/09/27 12:12:46 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:12:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:12:46 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (2 peer confirmations)
2021/09/27 12:12:46 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:12:46 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 12:12:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:12:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52220->10.242.28.11:8301: i/o timeout
2021/09/27 12:12:49 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:12:49 [ERR] raft: Failed to pipeline AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: write tcp 10.242.225.64:34608->10.242.28.11:8300: use of closed network connection
2021/09/27 12:12:49 [INFO] raft: aborting pipeline replication to peer {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}
2021/09/27 12:12:50 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:12:50 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 12:12:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:12:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:49612->10.242.28.28:8301: i/o timeout
2021/09/27 12:12:52 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:12:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:12:53 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500298565s
2021/09/27 12:12:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:12:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52330->10.242.28.11:8301: i/o timeout
2021/09/27 12:12:55 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:12:55 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.975438711s
2021/09/27 12:12:56 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (2 peer confirmations)
2021/09/27 12:12:56 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:12:56 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 12:12:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:12:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52394->10.242.28.11:8301: i/o timeout
2021/09/27 12:12:57 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:12:58 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500303653s
2021/09/27 12:12:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:12:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52418->10.242.28.11:8301: i/o timeout
2021/09/27 12:12:59 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:12:59 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:36090->10.242.28.11:8300: i/o timeout
2021/09/27 12:13:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:00 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.963287482s
2021/09/27 12:13:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52490->10.242.28.11:8301: i/o timeout
2021/09/27 12:13:01 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:13:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52534->10.242.28.11:8301: i/o timeout
2021/09/27 12:13:03 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:13:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52560->10.242.28.11:8301: i/o timeout
2021/09/27 12:13:05 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:13:05 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500323378s
2021/09/27 12:13:06 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations)
2021/09/27 12:13:06 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.28.11
2021/09/27 12:13:06 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 12:13:06 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/09/27 12:13:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:07 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:13:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:49054->10.242.28.11:8302: i/o timeout
2021/09/27 12:13:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:09 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500372103s
2021/09/27 12:13:09 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:36318->10.242.28.11:8300: i/o timeout
2021/09/27 12:13:09 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:13:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:11 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.28.11
2021/09/27 12:13:11 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 12:13:11 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/09/27 12:13:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52736->10.242.28.11:8301: i/o timeout
2021/09/27 12:13:13 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:13:13 [ERR] yamux: keepalive failed: i/o deadline reached
2021/09/27 12:13:13 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.28.28:39698
2021/09/27 12:13:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:15 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500308403s
2021/09/27 12:13:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52784->10.242.28.11:8301: i/o timeout
2021/09/27 12:13:16 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:13:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52808->10.242.28.11:8301: i/o timeout
2021/09/27 12:13:18 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:13:18 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.934613741s
2021/09/27 12:13:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:19 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:36522->10.242.28.11:8300: i/o timeout
2021/09/27 12:13:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52876->10.242.28.11:8301: i/o timeout
2021/09/27 12:13:20 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:13:20 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 7.406717026s
2021/09/27 12:13:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52920->10.242.28.11:8301: i/o timeout
2021/09/27 12:13:22 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:13:23 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:13:24 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500308925s
2021/09/27 12:13:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52986->10.242.28.11:8301: i/o timeout
2021/09/27 12:13:25 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:13:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:27 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.974764037s
2021/09/27 12:13:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:49468->10.242.28.11:8302: i/o timeout
2021/09/27 12:13:28 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:13:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:53036->10.242.28.11:8301: i/o timeout
2021/09/27 12:13:28 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:13:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:29 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 7.461842062s
2021/09/27 12:13:29 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:36728->10.242.28.11:8300: i/o timeout
2021/09/27 12:13:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:53062->10.242.28.11:8301: i/o timeout
2021/09/27 12:13:30 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:13:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:31 [ERR] memberlist: failed to receive: read tcp 10.242.225.64:8301->10.242.28.11:37958: i/o timeout from=10.242.28.11:37958
2021/09/27 12:13:31 [ERR] memberlist: Failed to send error: write tcp 10.242.225.64:8301->10.242.28.11:37958: i/o timeout from=10.242.28.11:37958
2021/09/27 12:13:32 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:34640->10.242.28.11:8300: i/o timeout
2021/09/27 12:13:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:53148->10.242.28.11:8301: i/o timeout
2021/09/27 12:13:33 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:13:34 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations)
2021/09/27 12:13:34 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.28.11
2021/09/27 12:13:34 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 12:13:34 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/09/27 12:13:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:53194->10.242.28.11:8301: i/o timeout
2021/09/27 12:13:35 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:13:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:49688->10.242.28.11:8302: i/o timeout
2021/09/27 12:13:38 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:13:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:39 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:36936->10.242.28.11:8300: i/o timeout
2021/09/27 12:13:40 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:13:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:43 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:37014->10.242.28.11:8300: i/o timeout
2021/09/27 12:13:44 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:13:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:46 [INFO] serf: attempting reconnect to dev-consul-68b98449d-z4w69 10.242.28.28:8301
2021/09/27 12:13:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:49890->10.242.28.11:8302: i/o timeout
2021/09/27 12:13:48 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:13:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:50 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:37140->10.242.28.11:8300: i/o timeout
2021/09/27 12:13:50 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:13:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:51 [ERR] memberlist: Push/Pull with dev-consul-server-0.dc1 failed: read tcp 10.242.225.64:49386->10.242.28.11:8302: i/o timeout
2021/09/27 12:13:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:54 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:37232->10.242.28.11:8300: i/o timeout
2021/09/27 12:13:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:54 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:13:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:13:57 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.28.11
2021/09/27 12:13:57 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/09/27 12:13:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:00 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:37346->10.242.28.11:8300: i/o timeout
2021/09/27 12:14:00 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:14:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:01 [ERR] yamux: keepalive failed: i/o deadline reached
2021/09/27 12:14:01 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.28.11:54794
2021/09/27 12:14:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:04 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:37448->10.242.28.11:8300: i/o timeout
2021/09/27 12:14:05 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:14:05 [ERR] yamux: keepalive failed: i/o deadline reached
2021/09/27 12:14:05 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/09/27 12:14:07 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:14:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:10 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:37554->10.242.28.11:8300: i/o timeout
2021/09/27 12:14:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:11 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:14:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:15 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:37662->10.242.28.11:8300: i/o timeout
2021/09/27 12:14:16 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:14:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:21 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:37770->10.242.28.11:8300: i/o timeout
2021/09/27 12:14:22 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:14:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:26 [INFO] serf: attempting reconnect to dev-consul-68b98449d-z4w69 10.242.28.28:8301
2021/09/27 12:14:26 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:37888->10.242.28.11:8300: i/o timeout
2021/09/27 12:14:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:27 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:14:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:32 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:37996->10.242.28.11:8300: i/o timeout
2021/09/27 12:14:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:33 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.28.11:8302
2021/09/27 12:14:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:35 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:14:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:37 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:38108->10.242.28.11:8300: i/o timeout
2021/09/27 12:14:38 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:14:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:45 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:38256->10.242.28.11:8300: i/o timeout
2021/09/27 12:14:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:46 [ERR] yamux: keepalive failed: i/o deadline reached
2021/09/27 12:14:46 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/09/27 12:14:48 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:38334->10.242.28.11:8300: i/o timeout
2021/09/27 12:14:49 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:14:49 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:14:49 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/09/27 12:14:49 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:14:50 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:14:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:57 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/09/27 12:14:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:14:59 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:14:59 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 12:14:59 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:38562->10.242.28.11:8300: i/o timeout
2021/09/27 12:15:00 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:38570->10.242.28.11:8300: i/o timeout
2021/09/27 12:15:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52242->10.242.28.28:8301: i/o timeout
2021/09/27 12:15:01 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:15:01 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:15:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52288->10.242.28.28:8301: i/o timeout
2021/09/27 12:15:03 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:15:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52334->10.242.28.28:8301: i/o timeout
2021/09/27 12:15:05 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:15:05 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (0 peer confirmations)
2021/09/27 12:15:05 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:15:05 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 12:15:06 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.28.11:8301
2021/09/27 12:15:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:09 [ERR] memberlist: failed to receive: read tcp 10.242.225.64:8301->10.242.28.28:50448: i/o timeout from=10.242.28.28:50448
2021/09/27 12:15:09 [ERR] memberlist: Failed to send error: write tcp 10.242.225.64:8301->10.242.28.28:50448: i/o timeout from=10.242.28.28:50448
2021/09/27 12:15:10 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:15:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:11 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:38788->10.242.28.11:8300: i/o timeout
2021/09/27 12:15:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:15 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:15:16 [ERR] memberlist: failed to receive: read tcp 10.242.225.64:8302->10.242.28.11:50636: i/o timeout from=10.242.28.11:50636
2021/09/27 12:15:16 [ERR] memberlist: Failed to send error: write tcp 10.242.225.64:8302->10.242.28.11:50636: i/o timeout from=10.242.28.11:50636
2021/09/27 12:15:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:20 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:38960->10.242.28.11:8300: i/o timeout
2021/09/27 12:15:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:25 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:39062->10.242.28.11:8300: i/o timeout
2021/09/27 12:15:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:28 [ERR] yamux: keepalive failed: i/o deadline reached
2021/09/27 12:15:28 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/09/27 12:15:30 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:15:31 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:15:31 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:15:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:33 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.28.11:8302
2021/09/27 12:15:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:38 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:15:38 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 12:15:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:53020->10.242.28.28:8301: i/o timeout
2021/09/27 12:15:40 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:15:40 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:39340->10.242.28.11:8300: i/o timeout
2021/09/27 12:15:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:41 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:39358->10.242.28.11:8300: i/o timeout
2021/09/27 12:15:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:53066->10.242.28.28:8301: i/o timeout
2021/09/27 12:15:43 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:15:44 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (0 peer confirmations)
2021/09/27 12:15:44 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:15:44 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 12:15:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:53110->10.242.28.28:8301: i/o timeout
2021/09/27 12:15:45 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:15:46 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.28.11:8301
2021/09/27 12:15:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:50 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:15:52 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:15:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:53 [WARN] memberlist: Refuting a suspect message (from: dev-consul-68b98449d-z4w69)
2021/09/27 12:15:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:15:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:00 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:39742->10.242.28.11:8300: i/o timeout
2021/09/27 12:16:02 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:39748->10.242.28.11:8300: i/o timeout
2021/09/27 12:16:02 [ERR] memberlist: failed to receive: read tcp 10.242.225.64:8301->10.242.28.28:51084: i/o timeout from=10.242.28.28:51084
2021/09/27 12:16:02 [ERR] memberlist: Failed to send error: write tcp 10.242.225.64:8301->10.242.28.28:51084: i/o timeout from=10.242.28.28:51084
2021/09/27 12:16:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:10 [ERR] yamux: keepalive failed: i/o deadline reached
2021/09/27 12:16:10 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/09/27 12:16:11 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:16:13 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:16:13 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:16:13 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:16:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:21 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:39796->10.242.28.11:8300: i/o timeout
2021/09/27 12:16:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:23 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:39806->10.242.28.11:8300: i/o timeout
2021/09/27 12:16:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:26 [INFO] serf: attempting reconnect to dev-consul-68b98449d-z4w69 10.242.28.28:8301
2021/09/27 12:16:26 [ERR] yamux: keepalive failed: i/o deadline reached
2021/09/27 12:16:26 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.28.11:58408
2021/09/27 12:16:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:31 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:16:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:34 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:16:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:41 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:39852->10.242.28.11:8300: i/o timeout
2021/09/27 12:16:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:44 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:39858->10.242.28.11:8300: i/o timeout
2021/09/27 12:16:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:47 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.28.11
2021/09/27 12:16:47 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 12:16:47 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/09/27 12:16:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:56044->10.242.28.11:8301: i/o timeout
2021/09/27 12:16:51 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:16:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:56048->10.242.28.11:8301: i/o timeout
2021/09/27 12:16:53 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:16:53 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (0 peer confirmations)
2021/09/27 12:16:53 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.28.11
2021/09/27 12:16:53 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 12:16:53 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/09/27 12:16:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:55 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:16:55 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.28.11
2021/09/27 12:16:55 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/09/27 12:16:55 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:16:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:16:57 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500342456s
2021/09/27 12:16:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:00 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.973606628s
2021/09/27 12:17:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:01 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:39892->10.242.28.11:8300: i/o timeout
2021/09/27 12:17:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:02 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 7.42037521s
2021/09/27 12:17:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52534->10.242.28.11:8302: i/o timeout
2021/09/27 12:17:03 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:17:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:05 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:39902->10.242.28.11:8300: i/o timeout
2021/09/27 12:17:06 [INFO] serf: attempting reconnect to dev-consul-68b98449d-z4w69 10.242.28.28:8301
2021/09/27 12:17:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:06 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:17:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:11 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:17:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:16 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:39928->10.242.28.11:8300: i/o timeout
2021/09/27 12:17:17 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:17:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52556->10.242.28.11:8302: i/o timeout
2021/09/27 12:17:18 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:17:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:19 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.28.11
2021/09/27 12:17:19 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 12:17:19 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/09/27 12:17:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:21 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/09/27 12:17:21 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:39942->10.242.28.11:8300: i/o timeout
2021/09/27 12:17:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:56128->10.242.28.11:8301: i/o timeout
2021/09/27 12:17:24 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:17:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:26 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.28.11
2021/09/27 12:17:26 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 12:17:26 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/09/27 12:17:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:56136->10.242.28.11:8301: i/o timeout
2021/09/27 12:17:27 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:17:27 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500293754s
2021/09/27 12:17:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:30 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.975491766s
2021/09/27 12:17:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:32 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:17:32 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 7.437045437s
2021/09/27 12:17:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52608->10.242.28.11:8302: i/o timeout
2021/09/27 12:17:33 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:17:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:35 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:39954->10.242.28.11:8300: i/o timeout
2021/09/27 12:17:36 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:17:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:42 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:39994->10.242.28.11:8300: i/o timeout
2021/09/27 12:17:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52630->10.242.28.11:8302: i/o timeout
2021/09/27 12:17:43 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:17:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:46 [INFO] serf: attempting reconnect to dev-consul-68b98449d-z4w69 10.242.28.28:8301
2021/09/27 12:17:46 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:40002->10.242.28.11:8300: i/o timeout
2021/09/27 12:17:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:47 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:17:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:52 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:17:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:53 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:17:54 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:17:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:55 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.28.11
2021/09/27 12:17:55 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 12:17:55 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/09/27 12:17:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:56206->10.242.28.11:8301: i/o timeout
2021/09/27 12:17:57 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:17:58 [WARN] memberlist: Was able to connect to dev-consul-server-0.dc1 but other probes failed, network may be misconfigured
2021/09/27 12:17:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:17:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:56220->10.242.28.11:8301: i/o timeout
2021/09/27 12:17:59 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:18:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:18:01 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/09/27 12:18:01 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (0 peer confirmations)
2021/09/27 12:18:01 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.28.11
2021/09/27 12:18:01 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 12:18:01 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/09/27 12:18:02 [ERR] yamux: keepalive failed: i/o deadline reached
2021/09/27 12:18:02 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.28.28:42960
2021/09/27 12:18:02 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500396752s
2021/09/27 12:18:02 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:40040->10.242.28.11:8300: i/o timeout
2021/09/27 12:18:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:18:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:18:04 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.944516676s
2021/09/27 12:18:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:18:07 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 7.377755528s
2021/09/27 12:18:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52692->10.242.28.11:8302: i/o timeout
2021/09/27 12:18:08 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:18:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:18:09 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.28.11
2021/09/27 12:18:09 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 12:18:09 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/09/27 12:18:12 [INFO] raft: pipelining replication to peer {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}
2021/09/27 12:18:13 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:18:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:18:23 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:18:35 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:18:35 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:18:37 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:18:37 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:18:37 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 12:18:39 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:18:39 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:18:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:53652->10.242.28.28:8301: i/o timeout
2021/09/27 12:18:40 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:18:40 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500278351s
2021/09/27 12:18:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:18:42 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.981538699s
2021/09/27 12:18:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:18:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52790->10.242.28.11:8302: i/o timeout
2021/09/27 12:18:43 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:18:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:56328->10.242.28.11:8301: i/o timeout
2021/09/27 12:18:43 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:18:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:18:45 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500619265s
2021/09/27 12:18:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:53666->10.242.28.28:8301: i/o timeout
2021/09/27 12:18:46 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:18:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:18:47 [INFO] raft: aborting pipeline replication to peer {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}
2021/09/27 12:18:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:56344->10.242.28.11:8301: i/o timeout
2021/09/27 12:18:48 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:18:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:18:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:53684->10.242.28.28:8301: i/o timeout
2021/09/27 12:18:50 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:18:50 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500241524s
2021/09/27 12:18:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:18:51 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (1 peer confirmations)
2021/09/27 12:18:51 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:18:51 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 12:18:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:56388->10.242.28.11:8301: i/o timeout
2021/09/27 12:18:52 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:18:52 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations)
2021/09/27 12:18:52 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.28.11
2021/09/27 12:18:52 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 12:18:52 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/09/27 12:18:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:18:52 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.929061092s
2021/09/27 12:18:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52850->10.242.28.11:8302: i/o timeout
2021/09/27 12:18:53 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:18:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:18:55 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 7.42898298s
2021/09/27 12:18:56 [INFO] serf: attempting reconnect to dev-consul-68b98449d-z4w69 10.242.28.28:8301
2021/09/27 12:18:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:18:56 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:40096->10.242.28.11:8300: i/o timeout
2021/09/27 12:18:57 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:18:58 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:40194->10.242.28.11:8300: i/o timeout
2021/09/27 12:18:58 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:18:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:02 [ERR] yamux: keepalive failed: i/o deadline reached
2021/09/27 12:19:02 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.28.11:59350
2021/09/27 12:19:02 [ERR] yamux: keepalive failed: i/o deadline reached
2021/09/27 12:19:02 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/09/27 12:19:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:52882->10.242.28.11:8302: i/o timeout
2021/09/27 12:19:03 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:19:03 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:19:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:07 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:40296->10.242.28.11:8300: i/o timeout
2021/09/27 12:19:08 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:40300->10.242.28.11:8300: i/o timeout
2021/09/27 12:19:08 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:19:08 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:19:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:53062->10.242.28.11:8302: i/o timeout
2021/09/27 12:19:18 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:19:18 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:40394->10.242.28.11:8300: i/o timeout
2021/09/27 12:19:18 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:19:18 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:40400->10.242.28.11:8300: i/o timeout
2021/09/27 12:19:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:19 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:19:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:21 [ERR] memberlist: Push/Pull with dev-consul-server-0.dc1 failed: read tcp 10.242.225.64:52854->10.242.28.11:8302: i/o timeout
2021/09/27 12:19:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:23 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/09/27 12:19:23 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.28.11
2021/09/27 12:19:23 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/09/27 12:19:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:53122->10.242.28.11:8302: i/o timeout
2021/09/27 12:19:28 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:19:28 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:40482->10.242.28.11:8300: i/o timeout
2021/09/27 12:19:28 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:19:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:29 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:40500->10.242.28.11:8300: i/o timeout
2021/09/27 12:19:30 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:19:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:32 [ERR] yamux: keepalive failed: i/o deadline reached
2021/09/27 12:19:32 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.28.28:44008
2021/09/27 12:19:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:36 [INFO] serf: attempting reconnect to dev-consul-68b98449d-z4w69 10.242.28.28:8301
2021/09/27 12:19:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:38 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:40532->10.242.28.11:8300: i/o timeout
2021/09/27 12:19:38 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:19:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:40 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:40552->10.242.28.11:8300: i/o timeout
2021/09/27 12:19:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:40 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:19:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:42 [ERR] yamux: keepalive failed: i/o deadline reached
2021/09/27 12:19:42 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/09/27 12:19:45 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:19:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:48 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:40590->10.242.28.11:8300: i/o timeout
2021/09/27 12:19:48 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:19:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:50 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:40600->10.242.28.11:8300: i/o timeout
2021/09/27 12:19:51 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:19:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:19:58 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:40632->10.242.28.11:8300: i/o timeout
2021/09/27 12:19:58 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:19:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:20:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:20:01 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:40646->10.242.28.11:8300: i/o timeout
2021/09/27 12:20:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:20:03 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:20:03 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.28.11:8302
2021/09/27 12:20:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:20:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:20:08 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: read tcp 10.242.225.64:40674->10.242.28.11:8300: i/o timeout
2021/09/27 12:20:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:20:09 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:20:10 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.28.11
2021/09/27 12:20:10 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 12:20:10 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/09/27 12:20:10 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.28.11
2021/09/27 12:20:10 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/09/27 12:20:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:20:13 [INFO] raft: pipelining replication to peer {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}
2021/09/27 12:20:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:56906->10.242.28.11:8301: i/o timeout
2021/09/27 12:20:16 [INFO] serf: attempting reconnect to dev-consul-68b98449d-z4w69 10.242.28.28:8301
2021/09/27 12:20:23 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:20:23 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 12:20:27 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:20:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:54296->10.242.28.28:8301: i/o timeout
2021/09/27 12:20:28 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:20:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:20:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:20:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:54314->10.242.28.28:8301: i/o timeout
2021/09/27 12:20:32 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:20:33 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:20:33 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 12:20:33 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:20:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:54324->10.242.28.28:8301: i/o timeout
2021/09/27 12:20:34 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:20:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:20:35 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500285057s
2021/09/27 12:20:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:20:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:56998->10.242.28.11:8301: i/o timeout
2021/09/27 12:20:37 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:20:37 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.920209885s
2021/09/27 12:20:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:53468->10.242.28.11:8302: i/o timeout
2021/09/27 12:20:38 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:20:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:20:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:57020->10.242.28.11:8301: i/o timeout
2021/09/27 12:20:40 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:20:40 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 7.335621409s
2021/09/27 12:20:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:20:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:57024->10.242.28.11:8301: i/o timeout
2021/09/27 12:20:42 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:20:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:20:42 [INFO] raft: aborting pipeline replication to peer {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}
2021/09/27 12:20:43 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:40688->10.242.28.11:8300: i/o timeout
2021/09/27 12:20:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:20:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:57042->10.242.28.11:8301: i/o timeout
2021/09/27 12:20:45 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:20:45 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations)
2021/09/27 12:20:45 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.28.11
2021/09/27 12:20:45 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 12:20:45 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/09/27 12:20:45 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.50033062s
2021/09/27 12:20:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:20:47 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.922426725s
2021/09/27 12:20:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:53512->10.242.28.11:8302: i/o timeout
2021/09/27 12:20:48 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:20:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:20:50 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.28.11
2021/09/27 12:20:50 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 12:20:50 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/09/27 12:20:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:20:51 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:20:51 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 12:20:52 [INFO] raft: pipelining replication to peer {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}
2021/09/27 12:20:57 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:20:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:20:59 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500316131s
2021/09/27 12:21:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:21:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:57104->10.242.28.11:8301: i/o timeout
2021/09/27 12:21:01 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:21:01 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.906525132s
2021/09/27 12:21:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:21:03 [WARN] memberlist: Was able to connect to dev-consul-68b98449d-z4w69 but other probes failed, network may be misconfigured
2021/09/27 12:21:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:57120->10.242.28.11:8301: i/o timeout
2021/09/27 12:21:04 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:21:04 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 7.36339014s
2021/09/27 12:21:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:21:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:54456->10.242.28.28:8301: i/o timeout
2021/09/27 12:21:07 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:21:07 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:21:07 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:21:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:21:09 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.50022058s
2021/09/27 12:21:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:57146->10.242.28.11:8301: i/o timeout
2021/09/27 12:21:10 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:21:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:21:11 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.941257082s
2021/09/27 12:21:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:21:13 [WARN] memberlist: Was able to connect to dev-consul-68b98449d-z4w69 but other probes failed, network may be misconfigured
2021/09/27 12:21:14 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 7.39458433s
2021/09/27 12:21:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:21:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:57174->10.242.28.11:8301: i/o timeout
2021/09/27 12:21:15 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:21:16 [INFO] raft: aborting pipeline replication to peer {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}
2021/09/27 12:21:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:21:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:57194->10.242.28.11:8301: i/o timeout
2021/09/27 12:21:17 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:21:17 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: read tcp 10.242.225.64:40882->10.242.28.11:8300: i/o timeout
2021/09/27 12:21:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:53652->10.242.28.11:8302: i/o timeout
2021/09/27 12:21:18 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:21:18 [INFO] memberlist: Marking dev-consul-68b98449d-z4w69 as failed, suspect timeout reached (1 peer confirmations)
2021/09/27 12:21:18 [INFO] serf: EventMemberFailed: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:21:18 [INFO] consul: member 'dev-consul-68b98449d-z4w69' failed, marking health critical
2021/09/27 12:21:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:21:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:54536->10.242.28.28:8301: i/o timeout
2021/09/27 12:21:19 [INFO] memberlist: Suspect dev-consul-68b98449d-z4w69 has failed, no acks received
2021/09/27 12:21:19 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500395524s
2021/09/27 12:21:20 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/09/27 12:21:20 [WARN] memberlist: Refuting a suspect message (from: dev-consul-68b98449d-z4w69)
2021/09/27 12:21:20 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:21:20 [INFO] consul: member 'dev-consul-68b98449d-z4w69' joined, marking health alive
2021/09/27 12:21:20 [INFO] raft: pipelining replication to peer {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}
2021/09/27 12:21:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.225.64:54552->10.242.28.28:8301: i/o timeout
2021/09/27 12:21:50 [INFO] serf: EventMemberLeave: dev-consul-68b98449d-z4w69 10.242.28.28
2021/09/27 12:21:50 [INFO] consul: member 'dev-consul-68b98449d-z4w69' left, deregistering
2021/09/27 12:21:51 [INFO] raft: aborting pipeline replication to peer {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}
2021/09/27 12:21:51 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: EOF
2021/09/27 12:21:51 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:21:51 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:21:51 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:21:51 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:21:51 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:21:51 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:21:52 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:21:52 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:21:52 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:21:52 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: stream closed
2021/09/27 12:21:52 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:21:53 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:21:53 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:21:53 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:21:53 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:21:53 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 2.500318561s
2021/09/27 12:21:54 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:21:54 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:21:54 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:21:54 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:21:55 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:21:55 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:21:56 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:21:56 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 4.955496568s
2021/09/27 12:21:56 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:21:56 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:21:57 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:21:57 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:21:57 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:21:58 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:21:58 [WARN] raft: Failed to contact 8262321d-b29f-83df-f962-bd890d88abc1 in 7.388515725s
2021/09/27 12:21:58 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:21:59 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:21:59 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:22:00 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:22:00 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:22:01 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:22:01 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:22:02 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:22:02 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:22:02 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.28.11:8300: getsockopt: connection refused
2021/09/27 12:22:03 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:22:05 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations)
2021/09/27 12:22:05 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.28.11
2021/09/27 12:22:05 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.28.11:8300) (DC: dc1)
2021/09/27 12:22:05 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/09/27 12:22:05 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:22:05 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:22:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:22:08 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:22:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:22:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:22:12 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:22:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:22:14 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/09/27 12:22:14 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: dial tcp ->10.242.28.11:8300: i/o timeout
2021/09/27 12:22:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:22:14 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.28.11:8300: i/o timeout
2021/09/27 12:22:16 [INFO] serf: EventMemberJoin: dev-consul-68b98449d-9l6r2 10.242.90.211
2021/09/27 12:22:16 [INFO] consul: member 'dev-consul-68b98449d-9l6r2' joined, marking health alive
2021/09/27 12:22:17 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:22:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:22:20 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:22:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:22:22 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: dial tcp ->10.242.28.11:8300: i/o timeout
2021/09/27 12:22:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:22:23 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.28.11:8301
2021/09/27 12:22:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:22:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:22:26 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.28.11:8300: i/o timeout
2021/09/27 12:22:28 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/09/27 12:22:28 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.28.11
2021/09/27 12:22:28 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/09/27 12:22:29 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:22:30 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: dial tcp ->10.242.28.11:8300: i/o timeout
2021/09/27 12:22:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:22:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:22:32 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:22:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:22:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:22:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:22:38 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.28.11:8300: i/o timeout
2021/09/27 12:22:41 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:22:41 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:22:41 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:22:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:22:42 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: dial tcp ->10.242.28.11:8300: i/o timeout
2021/09/27 12:22:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:22:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:22:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:22:50 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.28.11:8300: i/o timeout
2021/09/27 12:22:51 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/09/27 12:22:51 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: dial tcp ->10.242.28.11:8300: i/o timeout
2021/09/27 12:22:51 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:22:51 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/09/27 12:22:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:22:53 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:22:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:22:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:22:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:23:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/09/27 12:23:00 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.28.11:8300: i/o timeout
2021/09/27 12:23:02 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:23:03 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.28.11:8301
2021/09/27 12:23:03 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: dial tcp ->10.242.28.11:8300: i/o timeout
2021/09/27 12:23:03 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/09/27 12:23:04 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.225.79
2021/09/27 12:23:04 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.225.79:8300) (DC: dc1)
2021/09/27 12:23:04 [INFO] raft: Updating configuration with AddNonvoter (77f1a4bf-4bb0-7db1-a1ac-35e53b7597fb, 10.242.225.79:8300) to [{Suffrage:Voter ID:c466abc0-d439-190d-61b3-a6edbb2267f8 Address:10.242.225.64:8300} {Suffrage:Voter ID:8262321d-b29f-83df-f962-bd890d88abc1 Address:10.242.28.11:8300} {Suffrage:Voter ID:0044e147-4efb-f870-c8eb-6a34a13674cf Address:10.242.59.64:8300} {Suffrage:Nonvoter ID:77f1a4bf-4bb0-7db1-a1ac-35e53b7597fb Address:10.242.225.79:8300}]
2021/09/27 12:23:04 [INFO] raft: Added peer 77f1a4bf-4bb0-7db1-a1ac-35e53b7597fb, starting replication
2021/09/27 12:23:04 [WARN] raft: AppendEntries to {Nonvoter 77f1a4bf-4bb0-7db1-a1ac-35e53b7597fb 10.242.225.79:8300} rejected, sending older logs (next: 1)
2021/09/27 12:23:04 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/09/27 12:23:04 [ERR] memberlist: Conflicting address for dev-consul-server-0.dc1. Mine: 10.242.28.11:8302 Theirs: 10.242.225.79:8302
2021/09/27 12:23:04 [WARN] serf: Name conflict for 'dev-consul-server-0.dc1' both 10.242.28.11:8302 and 10.242.225.79:8302 are claiming
2021/09/27 12:23:04 [ERR] memberlist: Conflicting address for dev-consul-server-0.dc1. Mine: 10.242.28.11:8302 Theirs: 10.242.225.79:8302
2021/09/27 12:23:04 [WARN] serf: Name conflict for 'dev-consul-server-0.dc1' both 10.242.28.11:8302 and 10.242.225.79:8302 are claiming
2021/09/27 12:23:04 [ERR] memberlist: Conflicting address for dev-consul-server-0.dc1. Mine: 10.242.28.11:8302 Theirs: 10.242.225.79:8302
2021/09/27 12:23:04 [WARN] serf: Name conflict for 'dev-consul-server-0.dc1' both 10.242.28.11:8302 and 10.242.225.79:8302 are claiming
2021/09/27 12:23:04 [INFO] raft: pipelining replication to peer {Nonvoter 77f1a4bf-4bb0-7db1-a1ac-35e53b7597fb 10.242.225.79:8300}
2021/09/27 12:23:05 [ERR] memberlist: Conflicting address for dev-consul-server-0.dc1. Mine: 10.242.28.11:8302 Theirs: 10.242.225.79:8302
2021/09/27 12:23:05 [WARN] serf: Name conflict for 'dev-consul-server-0.dc1' both 10.242.28.11:8302 and 10.242.225.79:8302 are claiming
2021/09/27 12:23:05 [ERR] memberlist: Conflicting address for dev-consul-server-0.dc1. Mine: 10.242.28.11:8302 Theirs: 10.242.225.79:8302
2021/09/27 12:23:05 [WARN] serf: Name conflict for 'dev-consul-server-0.dc1' both 10.242.28.11:8302 and 10.242.225.79:8302 are claiming
2021/09/27 12:23:10 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.28.11:8302
2021/09/27 12:23:12 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: dial tcp ->10.242.28.11:8300: i/o timeout
2021/09/27 12:23:12 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.28.11:8300: i/o timeout
2021/09/27 12:23:13 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:23:16 [INFO] autopilot: Promoting Server (ID: "77f1a4bf-4bb0-7db1-a1ac-35e53b7597fb" Address: "10.242.225.79:8300") to voter
2021/09/27 12:23:16 [INFO] raft: Updating configuration with AddStaging (77f1a4bf-4bb0-7db1-a1ac-35e53b7597fb, 10.242.225.79:8300) to [{Suffrage:Voter ID:c466abc0-d439-190d-61b3-a6edbb2267f8 Address:10.242.225.64:8300} {Suffrage:Voter ID:8262321d-b29f-83df-f962-bd890d88abc1 Address:10.242.28.11:8300} {Suffrage:Voter ID:0044e147-4efb-f870-c8eb-6a34a13674cf Address:10.242.59.64:8300} {Suffrage:Voter ID:77f1a4bf-4bb0-7db1-a1ac-35e53b7597fb Address:10.242.225.79:8300}]
2021/09/27 12:23:17 [INFO] autopilot: Attempting removal of stale Server (ID: "8262321d-b29f-83df-f962-bd890d88abc1" Address: "10.242.28.11:8300")
2021/09/27 12:23:17 [INFO] raft: Updating configuration with RemoveServer (8262321d-b29f-83df-f962-bd890d88abc1, ) to [{Suffrage:Voter ID:c466abc0-d439-190d-61b3-a6edbb2267f8 Address:10.242.225.64:8300} {Suffrage:Voter ID:0044e147-4efb-f870-c8eb-6a34a13674cf Address:10.242.59.64:8300} {Suffrage:Voter ID:77f1a4bf-4bb0-7db1-a1ac-35e53b7597fb Address:10.242.225.79:8300}]
2021/09/27 12:23:17 [INFO] raft: Removed peer 8262321d-b29f-83df-f962-bd890d88abc1, stopping replication after 1375
2021/09/27 12:23:23 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:23:23 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: dial tcp ->10.242.28.11:8300: i/o timeout
2021/09/27 12:23:33 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: dial tcp ->10.242.28.11:8300: i/o timeout
2021/09/27 12:23:33 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:23:43 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: dial tcp ->10.242.28.11:8300: i/o timeout
2021/09/27 12:23:43 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:23:53 [WARN] Unable to get address for server id 8262321d-b29f-83df-f962-bd890d88abc1, using fallback address 10.242.28.11:8300: Could not find address for server id 8262321d-b29f-83df-f962-bd890d88abc1
2021/09/27 12:23:53 [ERR] raft: Failed to heartbeat to 10.242.28.11:8300: dial tcp ->10.242.28.11:8300: i/o timeout
2021/09/27 12:24:03 [ERR] raft: Failed to AppendEntries to {Voter 8262321d-b29f-83df-f962-bd890d88abc1 10.242.28.11:8300}: dial tcp ->10.242.28.11:8300: i/o timeout
2021/09/27 12:24:10 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.28.11:8302
2021/09/27 12:24:31 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.225.79
2021/09/27 12:24:31 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"