By type
+ CONSUL_BIND=
+ [ -n ]
+ CONSUL_CLIENT=
+ [ -n ]
+ CONSUL_DATA_DIR=/consul/data
+ CONSUL_CONFIG_DIR=/consul/config
+ [ -n ]
+ [ a = - ]
+ [ agent = agent ]
+ shift
+ set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
+ [ consul = consul ]
+ stat -c %u /consul/data
+ id -u consul
+ [ 100 != 100 ]
+ stat -c %u /consul/config
+ id -u consul
+ [ 100 != 100 ]
+ [ ! -z ]
+ exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
bootstrap_expect > 0: expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
Version: 'v1.0.6'
Node ID: '89c1a268-ee85-dfa0-749a-f6802dc15803'
Node name: 'dev-consul-server-0'
Datacenter: 'dc1' (Segment: '')
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.242.189.8 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2021/04/01 23:14:57 [INFO] raft: Initial configuration (index=0): []
2021/04/01 23:14:57 [INFO] raft: Node at 10.242.189.8:8300 [Follower] entering Follower state (Leader: "")
2021/04/01 23:14:57 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.189.8
2021/04/01 23:14:57 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.189.8
2021/04/01 23:14:57 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.189.8:8300) (DC: dc1)
2021/04/01 23:14:57 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/04/01 23:14:57 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2021/04/01 23:14:57 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2021/04/01 23:14:57 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
2021/04/01 23:14:57 [INFO] agent: started state syncer
2021/04/01 23:14:57 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer
2021/04/01 23:14:57 [INFO] agent: Joining LAN cluster...
2021/04/01 23:14:57 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/04/01 23:14:57 [WARN] memberlist: Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
2021/04/01 23:14:57 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
2021/04/01 23:14:57 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/04/01 23:14:57 [INFO] agent: (LAN) joined: 0 Err: 3 error(s) occurred:
* Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/04/01 23:14:57 [WARN] agent: Join LAN failed: , retrying in 30s
2021/04/01 23:15:03 [WARN] raft: no known peers, aborting election
2021/04/01 23:15:04 [ERR] agent: failed to sync remote state: No cluster leader
==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=28984263-f52f-96af-51f7-e9ea2307c4fa&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2021/04/01 23:15:27 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/04/01 23:15:27 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
2021/04/01 23:15:27 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/04/01 23:15:27 [INFO] agent: (LAN) joined: 1 Err:
2021/04/01 23:15:27 [INFO] agent: Join LAN completed. Synced with 1 initial agents
2021/04/01 23:15:28 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/01 23:15:30 [ERR] agent: Coordinate update error: No cluster leader
2021/04/01 23:15:33 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-qzw6d 10.242.189.7
2021/04/01 23:15:47 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/01 23:15:47 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/01 23:15:47 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/01 23:15:47 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/01 23:15:56 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/01 23:16:02 [ERR] agent: Coordinate update error: No cluster leader
2021/04/01 23:16:11 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.51.203
2021/04/01 23:16:11 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.51.203:8300) (DC: dc1)
2021/04/01 23:16:11 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.242.189.8:8300,10.242.241.18:8300,10.242.51.203:8300
2021/04/01 23:16:11 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.51.203
2021/04/01 23:16:11 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/01 23:16:15 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/01 23:16:15 [INFO] raft: Node at 10.242.189.8:8300 [Candidate] entering Candidate state in term 2
2021/04/01 23:16:15 [INFO] raft: Election won. Tally: 2
2021/04/01 23:16:15 [INFO] raft: Node at 10.242.189.8:8300 [Leader] entering Leader state
2021/04/01 23:16:15 [INFO] raft: Added peer ab3e202d-9a1e-5268-25eb-4c4881c4e864, starting replication
2021/04/01 23:16:15 [INFO] raft: Added peer fa409b74-3c2a-bdc6-3a5b-335beb268113, starting replication
2021/04/01 23:16:15 [INFO] consul: cluster leadership acquired
2021/04/01 23:16:15 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/01 23:16:15 [INFO] raft: pipelining replication to peer {Voter fa409b74-3c2a-bdc6-3a5b-335beb268113 10.242.51.203:8300}
2021/04/01 23:16:15 [WARN] raft: AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300} rejected, sending older logs (next: 1)
2021/04/01 23:16:15 [INFO] raft: pipelining replication to peer {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}
2021/04/01 23:16:15 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/04/01 23:16:15 [INFO] consul: member 'dev-consul-7467bf7b46-qzw6d' joined, marking health alive
2021/04/01 23:16:15 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/01 23:16:15 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/01 23:16:15 [INFO] agent: Synced node info
2021/04/01 23:38:22 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/01 23:38:22 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/01 23:38:23 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500152103s
2021/04/01 23:38:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:38:24 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/04/01 23:38:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:38:27 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.5002663s
2021/04/01 23:38:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:38:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:57816->10.242.241.18:8301: i/o timeout
2021/04/01 23:38:27 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/01 23:38:27 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/01 23:38:27 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/01 23:38:27 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/01 23:38:27 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/01 23:38:28 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/01 23:38:28 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/01 23:38:28 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/01 23:38:29 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/04/01 23:38:46 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/01 23:38:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:38:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:39:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/01 23:39:20 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/01 23:39:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:39:21 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500277503s
2021/04/01 23:54:56 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/01 23:54:56 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/01 23:55:00 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/01 23:55:00 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/01 23:55:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:55:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:40940->10.242.241.18:8301: i/o timeout
2021/04/01 23:55:01 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/01 23:55:02 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500275489s
2021/04/01 23:55:04 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/01 23:55:10 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/01 23:55:10 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/01 23:55:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:41044->10.242.241.18:8301: i/o timeout
2021/04/01 23:55:10 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/01 23:55:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:55:11 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500249718s
2021/04/01 23:55:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:55:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:41088->10.242.241.18:8301: i/o timeout
2021/04/01 23:55:13 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/01 23:55:14 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/01 23:55:14 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/01 23:55:14 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/01 23:55:14 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/01 23:55:14 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:55:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:55:16 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:55:16 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/01 23:55:16 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/01 23:55:16 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/01 23:55:18 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/01 23:55:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/01 23:55:32 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/01 23:55:32 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/01 23:55:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:55:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:41326->10.242.241.18:8301: i/o timeout
2021/04/01 23:55:33 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/01 23:55:34 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500231145s
2021/04/01 23:55:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:55:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:41340->10.242.241.18:8301: i/o timeout
2021/04/01 23:55:36 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/01 23:55:36 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 4.99547184s
2021/04/01 23:55:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:55:37 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/01 23:55:37 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/01 23:55:37 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/01 23:55:37 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/01 23:55:39 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 7.489004861s
2021/04/01 23:55:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:55:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:55:41 [INFO] raft: aborting pipeline replication to peer {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}
2021/04/01 23:55:41 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:55:42 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:50372->10.242.241.18:8300: i/o timeout
2021/04/01 23:55:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:56600->10.242.241.18:8302: i/o timeout
2021/04/01 23:55:42 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/01 23:55:43 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:55:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:55:44 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500232107s
2021/04/01 23:55:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:55:46 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 4.986562787s
2021/04/01 23:55:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:55:49 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 7.484379547s
2021/04/01 23:55:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:55:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:55:51 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:46904->10.242.241.18:8300: i/o timeout
2021/04/01 23:55:51 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:55:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:56718->10.242.241.18:8302: i/o timeout
2021/04/01 23:55:52 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/01 23:55:53 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:46930->10.242.241.18:8300: i/o timeout
2021/04/01 23:55:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:55:53 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/01 23:55:53 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.241.18:45942
2021/04/01 23:55:53 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/01 23:55:53 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/01 23:55:53 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:55:56 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/01 23:55:56 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/01 23:55:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:55:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:01 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:47030->10.242.241.18:8300: i/o timeout
2021/04/01 23:56:01 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:56:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:56836->10.242.241.18:8302: i/o timeout
2021/04/01 23:56:02 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/01 23:56:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:03 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:47052->10.242.241.18:8300: i/o timeout
2021/04/01 23:56:04 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:56:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:08 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/04/01 23:56:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:11 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:47142->10.242.241.18:8300: i/o timeout
2021/04/01 23:56:11 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:56:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:56948->10.242.241.18:8302: i/o timeout
2021/04/01 23:56:12 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/01 23:56:12 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/01 23:56:12 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/01 23:56:12 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/01 23:56:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:14 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:47170->10.242.241.18:8300: i/o timeout
2021/04/01 23:56:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:15 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:56:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:17 [ERR] memberlist: failed to receive: read tcp 10.242.189.8:8301->10.242.241.18:49732: i/o timeout from=10.242.241.18:49732
2021/04/01 23:56:17 [ERR] memberlist: Failed to send error: write tcp 10.242.189.8:8301->10.242.241.18:49732: i/o timeout from=10.242.241.18:49732
2021/04/01 23:56:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:21 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:47266->10.242.241.18:8300: i/o timeout
2021/04/01 23:56:21 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:56:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:25 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:47322->10.242.241.18:8300: i/o timeout
2021/04/01 23:56:26 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:56:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:27 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/01 23:56:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:31 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:47396->10.242.241.18:8300: i/o timeout
2021/04/01 23:56:31 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:56:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:35 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/01 23:56:35 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/01 23:56:36 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:47448->10.242.241.18:8300: i/o timeout
2021/04/01 23:56:37 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:56:38 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/01 23:56:38 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/01 23:56:38 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/01 23:56:38 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/01 23:56:38 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/01 23:56:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:42158->10.242.241.18:8301: i/o timeout
2021/04/01 23:56:41 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/01 23:56:41 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:47516->10.242.241.18:8300: i/o timeout
2021/04/01 23:56:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:42190->10.242.241.18:8301: i/o timeout
2021/04/01 23:56:44 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/01 23:56:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:45 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/01 23:56:45 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/01 23:56:45 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/01 23:56:45 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/01 23:56:47 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:47574->10.242.241.18:8300: i/o timeout
2021/04/01 23:56:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:48 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:56:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:52 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:47646->10.242.241.18:8300: i/o timeout
2021/04/01 23:56:52 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:56:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:58 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:47712->10.242.241.18:8300: i/o timeout
2021/04/01 23:56:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:56:59 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:57:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:02 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:47768->10.242.241.18:8300: i/o timeout
2021/04/01 23:57:03 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:57:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:09 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:47844->10.242.241.18:8300: i/o timeout
2021/04/01 23:57:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:12 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:57:13 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:47890->10.242.241.18:8300: i/o timeout
2021/04/01 23:57:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:14 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:57:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:17 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/01 23:57:17 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/01 23:57:18 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/01 23:57:18 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.241.18:36726
2021/04/01 23:57:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/01 23:57:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:22 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:48006->10.242.241.18:8300: i/o timeout
2021/04/01 23:57:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:24 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:48032->10.242.241.18:8300: i/o timeout
2021/04/01 23:57:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:25 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:57:27 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:57:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:27 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/01 23:57:27 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/01 23:57:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:35 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:48162->10.242.241.18:8300: i/o timeout
2021/04/01 23:57:37 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:48184->10.242.241.18:8300: i/o timeout
2021/04/01 23:57:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:41 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:57:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:42 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:57:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:48 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/01 23:57:48 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/01 23:57:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:51 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:48344->10.242.241.18:8300: i/o timeout
2021/04/01 23:57:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:52 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:48372->10.242.241.18:8300: i/o timeout
2021/04/01 23:57:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:58230->10.242.241.18:8302: i/o timeout
2021/04/01 23:57:57 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/01 23:57:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:57:59 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/01 23:57:59 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/01 23:58:01 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:58:02 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/01 23:58:02 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/01 23:58:02 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:58:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:58352->10.242.241.18:8302: i/o timeout
2021/04/01 23:58:07 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/01 23:58:07 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/01 23:58:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:11 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:48592->10.242.241.18:8300: i/o timeout
2021/04/01 23:58:12 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:48600->10.242.241.18:8300: i/o timeout
2021/04/01 23:58:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:58536->10.242.241.18:8302: i/o timeout
2021/04/01 23:58:22 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/01 23:58:22 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:58:22 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:58:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:27 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/01 23:58:27 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/01 23:58:27 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/01 23:58:27 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/01 23:58:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:58654->10.242.241.18:8302: i/o timeout
2021/04/01 23:58:32 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/01 23:58:32 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:48852->10.242.241.18:8300: i/o timeout
2021/04/01 23:58:32 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:48860->10.242.241.18:8300: i/o timeout
2021/04/01 23:58:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:41 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/01 23:58:41 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/01 23:58:42 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:58:43 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:58:44 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/01 23:58:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:52 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:49092->10.242.241.18:8300: i/o timeout
2021/04/01 23:58:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:53 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:49104->10.242.241.18:8300: i/o timeout
2021/04/01 23:58:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:58:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:03 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:59:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:04 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:59:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:13 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:49322->10.242.241.18:8300: i/o timeout
2021/04/01 23:59:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:14 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:49334->10.242.241.18:8300: i/o timeout
2021/04/01 23:59:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:17 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/01 23:59:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:23 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/01 23:59:23 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/01 23:59:23 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:59:25 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:59:26 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/01 23:59:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/01 23:59:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:27 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/01 23:59:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:33 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:49566->10.242.241.18:8300: i/o timeout
2021/04/01 23:59:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:35 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:49598->10.242.241.18:8300: i/o timeout
2021/04/01 23:59:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:43 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:59:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:46 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/01 23:59:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:53 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:49812->10.242.241.18:8300: i/o timeout
2021/04/01 23:59:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:56 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:49846->10.242.241.18:8300: i/o timeout
2021/04/01 23:59:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/01 23:59:57 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/01 23:59:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:00:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:00:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:00:03 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:00:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:00:05 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:00:05 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:00:07 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:00:08 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:00:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/02 00:00:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:00:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:00:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:00:13 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:50036->10.242.241.18:8300: i/o timeout
2021/04/02 00:00:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:00:16 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:00:16 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:00:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:00:17 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:50086->10.242.241.18:8300: i/o timeout
2021/04/02 00:00:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:00:19 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:00:19 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:00:19 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/02 00:00:24 [INFO] raft: pipelining replication to peer {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}
2021/04/02 00:00:43 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/02 00:00:48 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:00:48 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/02 00:00:48 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.50020504s
2021/04/02 00:00:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:02:16 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:02:16 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/02 00:02:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:02:22 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:02:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:02:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:02:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:02:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:46254->10.242.241.18:8301: i/o timeout
2021/04/02 00:02:27 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:02:27 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500221857s
2021/04/02 00:02:30 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:02:30 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/02 00:02:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:46278->10.242.241.18:8301: i/o timeout
2021/04/02 00:02:30 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:02:31 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500196384s
2021/04/02 00:02:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:02:31 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:02:31 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:02:31 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:02:31 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/02 00:02:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:02:33 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 4.958653215s
2021/04/02 00:02:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:02:36 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:02:36 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:02:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:02:38 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.50022376s
2021/04/02 00:02:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:02:41 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 4.952458265s
2021/04/02 00:02:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:02:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:02:43 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 7.415510336s
2021/04/02 00:02:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:02:46 [ERR] raft: Failed to pipeline AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: write tcp 10.242.189.8:50296->10.242.241.18:8300: use of closed network connection
2021/04/02 00:02:46 [INFO] raft: aborting pipeline replication to peer {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}
2021/04/02 00:02:46 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:02:46 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:50342->10.242.241.18:8300: i/o timeout
2021/04/02 00:02:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:02:47 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:02:47 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:02:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:33428->10.242.241.18:8302: i/o timeout
2021/04/02 00:02:47 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:02:47 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:02:48 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500291069s
2021/04/02 00:02:50 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:02:51 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 4.970765763s
2021/04/02 00:02:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:02:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:02:53 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 7.435419105s
2021/04/02 00:02:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:02:56 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:51960->10.242.241.18:8300: i/o timeout
2021/04/02 00:02:56 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:02:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:02:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:33558->10.242.241.18:8302: i/o timeout
2021/04/02 00:02:57 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:02:57 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:51982->10.242.241.18:8300: i/o timeout
2021/04/02 00:02:58 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:02:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:00 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:03:00 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.241.18:39162
2021/04/02 00:03:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:06 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:52084->10.242.241.18:8300: i/o timeout
2021/04/02 00:03:06 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:03:06 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:03:06 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:03:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:33612->10.242.241.18:8302: i/o timeout
2021/04/02 00:03:07 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:03:08 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:52106->10.242.241.18:8300: i/o timeout
2021/04/02 00:03:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:09 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:03:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:16 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:52210->10.242.241.18:8300: i/o timeout
2021/04/02 00:03:16 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:03:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:19 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:52238->10.242.241.18:8300: i/o timeout
2021/04/02 00:03:20 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:03:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:26 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:52336->10.242.241.18:8300: i/o timeout
2021/04/02 00:03:26 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:03:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:27 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:03:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:29 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:03:29 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:03:30 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:52392->10.242.241.18:8300: i/o timeout
2021/04/02 00:03:31 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:03:32 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:03:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:36 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:52464->10.242.241.18:8300: i/o timeout
2021/04/02 00:03:36 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:03:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:41 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:52516->10.242.241.18:8300: i/o timeout
2021/04/02 00:03:41 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:03:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:46 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:52588->10.242.241.18:8300: i/o timeout
2021/04/02 00:03:46 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:03:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:51 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:52648->10.242.241.18:8300: i/o timeout
2021/04/02 00:03:52 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:03:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:56 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:52710->10.242.241.18:8300: i/o timeout
2021/04/02 00:03:56 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:03:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:03:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:02 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:52786->10.242.241.18:8300: i/o timeout
2021/04/02 00:04:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:04 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:04:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:06 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:52826->10.242.241.18:8300: i/o timeout
2021/04/02 00:04:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:07 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:04:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:11 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:04:11 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:04:14 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:52902->10.242.241.18:8300: i/o timeout
2021/04/02 00:04:14 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:04:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:16 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:04:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:17 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:52954->10.242.241.18:8300: i/o timeout
2021/04/02 00:04:18 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:04:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:26 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:53072->10.242.241.18:8300: i/o timeout
2021/04/02 00:04:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:28 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:53094->10.242.241.18:8300: i/o timeout
2021/04/02 00:04:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:29 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:04:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:31 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:04:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:37 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:04:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:39 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:53212->10.242.241.18:8300: i/o timeout
2021/04/02 00:04:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:41 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:53238->10.242.241.18:8300: i/o timeout
2021/04/02 00:04:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:45 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:04:46 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:04:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:53 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:04:53 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:04:55 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:53412->10.242.241.18:8300: i/o timeout
2021/04/02 00:04:56 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:04:56 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:53430->10.242.241.18:8300: i/o timeout
2021/04/02 00:04:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:04:57 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:04:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:06 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:05:06 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:05:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:16 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:53666->10.242.241.18:8300: i/o timeout
2021/04/02 00:05:16 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:53668->10.242.241.18:8300: i/o timeout
2021/04/02 00:05:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:27 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:05:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:27 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:05:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:35 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:05:35 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:05:37 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:53910->10.242.241.18:8300: i/o timeout
2021/04/02 00:05:37 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:53916->10.242.241.18:8300: i/o timeout
2021/04/02 00:05:38 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:05:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:47 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:05:48 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:05:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:05:57 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:54146->10.242.241.18:8300: i/o timeout
2021/04/02 00:05:57 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:05:58 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:54150->10.242.241.18:8300: i/o timeout
2021/04/02 00:05:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:07 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:06:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:09 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:06:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:17 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:06:17 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:06:17 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:06:17 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:54374->10.242.241.18:8300: i/o timeout
2021/04/02 00:06:19 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:54386->10.242.241.18:8300: i/o timeout
2021/04/02 00:06:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:06:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:27 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:06:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:30 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:06:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:37 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:54618->10.242.241.18:8300: i/o timeout
2021/04/02 00:06:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:40 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:54638->10.242.241.18:8300: i/o timeout
2021/04/02 00:06:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:48 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:06:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:51 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:06:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:58 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:54860->10.242.241.18:8300: i/o timeout
2021/04/02 00:06:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:06:59 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:06:59 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:07:01 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:54898->10.242.241.18:8300: i/o timeout
2021/04/02 00:07:02 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:07:02 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/02 00:07:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:08 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:07:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:12 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:07:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:18 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:55088->10.242.241.18:8300: i/o timeout
2021/04/02 00:07:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:22 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:55142->10.242.241.18:8300: i/o timeout
2021/04/02 00:07:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:27 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:07:28 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:07:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:33 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:07:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:38 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:55330->10.242.241.18:8300: i/o timeout
2021/04/02 00:07:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:41 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:07:41 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:07:43 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:55384->10.242.241.18:8300: i/o timeout
2021/04/02 00:07:44 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:07:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:48 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:07:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:54 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:07:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:07:58 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:55566->10.242.241.18:8300: i/o timeout
2021/04/02 00:07:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:04 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:55638->10.242.241.18:8300: i/o timeout
2021/04/02 00:08:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:09 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:08:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:15 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:08:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:19 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:55800->10.242.241.18:8300: i/o timeout
2021/04/02 00:08:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:23 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:08:23 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:08:25 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:55882->10.242.241.18:8300: i/o timeout
2021/04/02 00:08:26 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:08:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:27 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:08:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:29 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:08:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:35 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:08:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:39 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:56042->10.242.241.18:8300: i/o timeout
2021/04/02 00:08:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:45 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:56160->10.242.241.18:8300: i/o timeout
2021/04/02 00:08:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:49 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:08:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:56 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:08:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:57 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:08:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:08:59 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:56318->10.242.241.18:8300: i/o timeout
2021/04/02 00:09:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:05 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:09:05 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:09:06 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:56426->10.242.241.18:8300: i/o timeout
2021/04/02 00:09:08 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:09:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:09 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:09:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:17 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:09:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:19 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:56562->10.242.241.18:8300: i/o timeout
2021/04/02 00:09:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:27 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:09:27 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:56674->10.242.241.18:8300: i/o timeout
2021/04/02 00:09:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:30 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:09:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:38 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:09:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:40 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:56814->10.242.241.18:8300: i/o timeout
2021/04/02 00:09:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:47 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:09:47 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:09:48 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:56918->10.242.241.18:8300: i/o timeout
2021/04/02 00:09:50 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:09:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/02 00:09:50 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:09:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:09:59 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:10:00 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:57050->10.242.241.18:8300: i/o timeout
2021/04/02 00:10:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:09 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:57164->10.242.241.18:8300: i/o timeout
2021/04/02 00:10:10 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:10:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:20 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:57288->10.242.241.18:8300: i/o timeout
2021/04/02 00:10:20 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:10:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:29 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:10:29 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:10:30 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:57418->10.242.241.18:8300: i/o timeout
2021/04/02 00:10:30 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:10:32 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:10:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:37 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:10:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:40 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:57530->10.242.241.18:8300: i/o timeout
2021/04/02 00:10:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:41 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:10:42 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/02 00:10:43 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:10:43 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:10:43 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/02 00:10:43 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:10:43 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:10:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:52308->10.242.241.18:8301: i/o timeout
2021/04/02 00:10:48 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:10:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:51 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:10:51 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:10:51 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:10:51 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/02 00:10:51 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:10:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:51 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:57682->10.242.241.18:8300: i/o timeout
2021/04/02 00:10:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:52346->10.242.241.18:8301: i/o timeout
2021/04/02 00:10:51 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:10:53 [ERR] memberlist: Push/Pull with dev-consul-server-1 failed: read tcp 10.242.189.8:52248->10.242.241.18:8301: i/o timeout
2021/04/02 00:10:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:10:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:39372->10.242.241.18:8302: i/o timeout
2021/04/02 00:10:57 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:10:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:00 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:11:00 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.241.18:44682
2021/04/02 00:11:01 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:57816->10.242.241.18:8300: i/o timeout
2021/04/02 00:11:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:02 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:11:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:39492->10.242.241.18:8302: i/o timeout
2021/04/02 00:11:07 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:11:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:11 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:11:11 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:11:11 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:11:12 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:57976->10.242.241.18:8300: i/o timeout
2021/04/02 00:11:14 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:11:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:21 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:58066->10.242.241.18:8300: i/o timeout
2021/04/02 00:11:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:39664->10.242.241.18:8302: i/o timeout
2021/04/02 00:11:22 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:11:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:23 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:11:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:27 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/02 00:11:27 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:11:27 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:11:27 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:11:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:31 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:11:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:39810->10.242.241.18:8302: i/o timeout
2021/04/02 00:11:32 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:11:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:33 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:58228->10.242.241.18:8300: i/o timeout
2021/04/02 00:11:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:38 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:11:38 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:11:38 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/02 00:11:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:41 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:58344->10.242.241.18:8300: i/o timeout
2021/04/02 00:11:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:53038->10.242.241.18:8301: i/o timeout
2021/04/02 00:11:44 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:11:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:47 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:11:47 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:11:47 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:11:47 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/02 00:11:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:53042->10.242.241.18:8301: i/o timeout
2021/04/02 00:11:47 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:11:47 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:11:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:51 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:11:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:53 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:11:53 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:11:54 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:58508->10.242.241.18:8300: i/o timeout
2021/04/02 00:11:56 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:11:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:11:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:01 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:58594->10.242.241.18:8300: i/o timeout
2021/04/02 00:12:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:05 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:12:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:11 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:12:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:15 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:58760->10.242.241.18:8300: i/o timeout
2021/04/02 00:12:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:20 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:12:20 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:12:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:21 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:58848->10.242.241.18:8300: i/o timeout
2021/04/02 00:12:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:26 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:12:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:32 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:12:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:40528->10.242.241.18:8302: i/o timeout
2021/04/02 00:12:32 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:12:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:35 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:12:35 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:12:36 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:59016->10.242.241.18:8300: i/o timeout
2021/04/02 00:12:38 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:12:38 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/02 00:12:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:42 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:59086->10.242.241.18:8300: i/o timeout
2021/04/02 00:12:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:40600->10.242.241.18:8302: i/o timeout
2021/04/02 00:12:42 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:12:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:47 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:12:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:49 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:12:49 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.241.18:45440
2021/04/02 00:12:50 [ERR] memberlist: failed to receive: read tcp 10.242.189.8:8302->10.242.241.18:36366: i/o timeout from=10.242.241.18:36366
2021/04/02 00:12:50 [ERR] memberlist: Failed to send error: write tcp 10.242.189.8:8302->10.242.241.18:36366: i/o timeout from=10.242.241.18:36366
2021/04/02 00:12:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:52 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:12:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:40760->10.242.241.18:8302: i/o timeout
2021/04/02 00:12:52 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:12:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:54 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:12:54 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:12:54 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/02 00:12:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:56 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/02 00:12:56 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:12:56 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:12:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:12:57 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:59264->10.242.241.18:8300: i/o timeout
2021/04/02 00:12:57 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:12:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:53938->10.242.241.18:8301: i/o timeout
2021/04/02 00:13:00 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:13:00 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:13:00 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:13:00 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:13:00 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/02 00:13:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:02 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:59320->10.242.241.18:8300: i/o timeout
2021/04/02 00:13:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:08 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:13:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:12 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:13:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:17 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:13:17 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:13:18 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:59506->10.242.241.18:8300: i/o timeout
2021/04/02 00:13:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:13:20 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/02 00:13:20 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/02 00:13:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:22 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:59572->10.242.241.18:8300: i/o timeout
2021/04/02 00:13:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:29 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:13:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:32 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:13:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:39 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:59768->10.242.241.18:8300: i/o timeout
2021/04/02 00:13:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:42 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:59818->10.242.241.18:8300: i/o timeout
2021/04/02 00:13:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:50 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:13:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:53 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:13:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:57 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:13:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:13:59 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:13:59 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:14:00 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:60006->10.242.241.18:8300: i/o timeout
2021/04/02 00:14:02 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:14:02 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/02 00:14:03 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:60060->10.242.241.18:8300: i/o timeout
2021/04/02 00:14:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:11 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:14:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:13 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:14:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:21 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:60256->10.242.241.18:8300: i/o timeout
2021/04/02 00:14:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:23 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:60320->10.242.241.18:8300: i/o timeout
2021/04/02 00:14:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:27 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:14:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:32 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:14:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:33 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:14:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:41 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:14:41 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:14:42 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:60540->10.242.241.18:8300: i/o timeout
2021/04/02 00:14:43 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:60564->10.242.241.18:8300: i/o timeout
2021/04/02 00:14:44 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:14:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:52 [ERR] memberlist: failed to receive: read tcp 10.242.189.8:8302->10.242.241.18:37574: i/o timeout from=10.242.241.18:37574
2021/04/02 00:14:52 [ERR] memberlist: Failed to send error: write tcp 10.242.189.8:8302->10.242.241.18:37574: i/o timeout from=10.242.241.18:37574
2021/04/02 00:14:53 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:14:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:53 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:14:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:14:57 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:14:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:03 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:60792->10.242.241.18:8300: i/o timeout
2021/04/02 00:15:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:03 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:60808->10.242.241.18:8300: i/o timeout
2021/04/02 00:15:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:07 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:15:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:14 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:15:14 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:15:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:23 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:15:23 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:15:24 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:32824->10.242.241.18:8300: i/o timeout
2021/04/02 00:15:24 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:32826->10.242.241.18:8300: i/o timeout
2021/04/02 00:15:26 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:15:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:27 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:15:27 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:15:27 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/02 00:15:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:55814->10.242.241.18:8301: i/o timeout
2021/04/02 00:15:33 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:15:33 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:15:33 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:15:33 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:15:33 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/02 00:15:34 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:15:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:35 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:15:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:55828->10.242.241.18:8301: i/o timeout
2021/04/02 00:15:36 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:15:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:37 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/04/02 00:15:38 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:15:38 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:15:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:44 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:33068->10.242.241.18:8300: i/o timeout
2021/04/02 00:15:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:45 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:33078->10.242.241.18:8300: i/o timeout
2021/04/02 00:15:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:52 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/04/02 00:15:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:54 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:15:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:56 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:15:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:15:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:04 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:33314->10.242.241.18:8300: i/o timeout
2021/04/02 00:16:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:05 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:16:05 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:16:06 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:33334->10.242.241.18:8300: i/o timeout
2021/04/02 00:16:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:43150->10.242.241.18:8302: i/o timeout
2021/04/02 00:16:07 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:16:07 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:16:07 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.241.18:47728
2021/04/02 00:16:08 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:16:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/02 00:16:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:14 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:16:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:16 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/02 00:16:16 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:16:16 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:16:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:17 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:16:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:43290->10.242.241.18:8302: i/o timeout
2021/04/02 00:16:17 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:16:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:24 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:33568->10.242.241.18:8300: i/o timeout
2021/04/02 00:16:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:27 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:33596->10.242.241.18:8300: i/o timeout
2021/04/02 00:16:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:35 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:16:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:38 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:16:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:45 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.189.8:33804->10.242.241.18:8300: i/o timeout
2021/04/02 00:16:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:47 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:16:47 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:16:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:16:47 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:16:48 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.189.8:33840->10.242.241.18:8300: i/o timeout
2021/04/02 00:16:49 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:16:49 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:16:49 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/02 00:16:49 [WARN] memberlist: Refuting a dead message (from: dev-consul-server-1.dc1)
2021/04/02 00:16:54 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:16:54 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/02 00:16:55 [DEBUG] raft-net: 10.242.189.8:8300 accepted connection from: 10.242.241.18:49052
2021/04/02 00:16:55 [WARN] raft: Rejecting vote request from 10.242.241.18:8300 since we have a leader: 10.242.189.8:8300
2021/04/02 00:16:55 [ERR] raft: peer {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300} has newer term, stopping replication
2021/04/02 00:16:55 [INFO] raft: Node at 10.242.189.8:8300 [Follower] entering Follower state (Leader: "")
2021/04/02 00:16:55 [INFO] raft: aborting pipeline replication to peer {Voter fa409b74-3c2a-bdc6-3a5b-335beb268113 10.242.51.203:8300}
2021/04/02 00:16:55 [INFO] consul: cluster leadership lost
2021/04/02 00:16:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:56860->10.242.241.18:8301: i/o timeout
2021/04/02 00:16:59 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:17:03 [DEBUG] raft-net: 10.242.189.8:8300 accepted connection from: 10.242.51.203:39046
2021/04/02 00:17:03 [WARN] raft: Rejecting vote request from 10.242.241.18:8300 since we have a leader: 10.242.51.203:8300
2021/04/02 00:17:03 [INFO] consul: New leader elected: dev-consul-server-2
2021/04/02 00:17:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:56954->10.242.241.18:8301: i/o timeout
2021/04/02 00:17:06 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:17:10 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:17:10 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:17:10 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: rpc error making call: No cluster leader from=10.242.199.78:59266
2021/04/02 00:17:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:56990->10.242.241.18:8301: i/o timeout
2021/04/02 00:17:10 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:17:10 [WARN] raft: Rejecting vote request from 10.242.241.18:8300 since we have a leader: 10.242.51.203:8300
2021/04/02 00:17:10 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:17:10 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:17:13 [DEBUG] raft-net: 10.242.189.8:8300 accepted connection from: 10.242.51.203:39202
2021/04/02 00:17:13 [INFO] consul: New leader elected: dev-consul-server-2
2021/04/02 00:17:13 [DEBUG] raft-net: 10.242.189.8:8300 accepted connection from: 10.242.51.203:39210
2021/04/02 00:17:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:57070->10.242.241.18:8301: i/o timeout
2021/04/02 00:17:14 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:17:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:57146->10.242.241.18:8301: i/o timeout
2021/04/02 00:17:21 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:17:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:57268->10.242.241.18:8301: i/o timeout
2021/04/02 00:17:31 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:17:32 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:17:32 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:17:35 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:17:35 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:17:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:57358->10.242.241.18:8301: i/o timeout
2021/04/02 00:17:38 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:17:40 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:17:40 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:17:40 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:17:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:44336->10.242.241.18:8302: i/o timeout
2021/04/02 00:17:42 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:17:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:44452->10.242.241.18:8302: i/o timeout
2021/04/02 00:17:52 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:18:02 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:18:02 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/04/02 00:18:02 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:18:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:57682->10.242.241.18:8301: i/o timeout
2021/04/02 00:18:05 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:18:06 [WARN] raft: Rejecting vote request from 10.242.241.18:8300 since we have a leader: 10.242.51.203:8300
2021/04/02 00:18:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:44644->10.242.241.18:8302: i/o timeout
2021/04/02 00:18:07 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:18:09 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:18:09 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:18:09 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:18:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:57738->10.242.241.18:8301: i/o timeout
2021/04/02 00:18:09 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:18:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:44788->10.242.241.18:8302: i/o timeout
2021/04/02 00:18:22 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:18:28 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:18:28 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:18:28 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.241.18:48888
2021/04/02 00:18:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:44950->10.242.241.18:8302: i/o timeout
2021/04/02 00:18:32 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:18:37 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:18:37 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:18:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:58112->10.242.241.18:8301: i/o timeout
2021/04/02 00:18:39 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:18:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:45072->10.242.241.18:8302: i/o timeout
2021/04/02 00:18:42 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:18:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:58156->10.242.241.18:8301: i/o timeout
2021/04/02 00:18:42 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:18:43 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:18:43 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:18:43 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:18:51 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:18:51 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:18:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:45194->10.242.241.18:8302: i/o timeout
2021/04/02 00:18:52 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:20:14 [ERR] memberlist: failed to receive: read tcp 10.242.189.8:8302->10.242.241.18:41348: i/o timeout from=10.242.241.18:41348
2021/04/02 00:20:14 [ERR] memberlist: Failed to send error: write tcp 10.242.189.8:8302->10.242.241.18:41348: i/o timeout from=10.242.241.18:41348
2021/04/02 00:20:49 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:21:27 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:21:29 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:22:19 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:22:27 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:23:57 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:24:09 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:24:57 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:25:25 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:25:57 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:26:19 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:27:27 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:27:29 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:28:27 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:30:09 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:30:29 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:30:29 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:30:30 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:30:30 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:33:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:40864->10.242.241.18:8301: i/o timeout
2021/04/02 00:33:58 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:34:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:40974->10.242.241.18:8301: i/o timeout
2021/04/02 00:34:07 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:34:12 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/02 00:34:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:41074->10.242.241.18:8301: i/o timeout
2021/04/02 00:34:14 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:34:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:41118->10.242.241.18:8301: i/o timeout
2021/04/02 00:34:18 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:34:18 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:34:18 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:34:18 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:34:19 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:34:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:56326->10.242.241.18:8302: i/o timeout
2021/04/02 00:34:22 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:34:30 [ERR] yamux: keepalive failed: session shutdown
2021/04/02 00:34:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:56444->10.242.241.18:8302: i/o timeout
2021/04/02 00:34:32 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:34:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:56566->10.242.241.18:8302: i/o timeout
2021/04/02 00:34:42 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:34:51 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/02 00:34:51 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:34:51 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:35:27 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:36:29 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:36:57 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:37:39 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:38:57 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:40:27 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:40:49 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:41:27 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:41:29 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:42:27 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:43:09 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:43:35 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:43:35 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:43:35 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:43:35 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:43:36 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/02 00:43:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:47850->10.242.241.18:8301: i/o timeout
2021/04/02 00:43:41 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:43:42 [WARN] raft: Rejecting vote request from 10.242.241.18:8300 since we have a leader: 10.242.51.203:8300
2021/04/02 00:43:43 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:43:43 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:43:43 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:43:49 [INFO] consul: New leader elected: dev-consul-server-2
2021/04/02 00:43:50 [DEBUG] raft-net: 10.242.189.8:8300 accepted connection from: 10.242.51.203:35038
2021/04/02 00:43:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:34906->10.242.241.18:8302: i/o timeout
2021/04/02 00:43:52 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:44:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:35028->10.242.241.18:8302: i/o timeout
2021/04/02 00:44:02 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:44:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:35146->10.242.241.18:8302: i/o timeout
2021/04/02 00:44:12 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:44:15 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:44:15 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.241.18:34944
2021/04/02 00:44:19 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:44:21 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/02 00:44:21 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:44:21 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:44:27 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:44:29 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:44:29 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:44:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:48464->10.242.241.18:8301: i/o timeout
2021/04/02 00:44:31 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:44:35 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:44:35 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:44:35 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:44:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:48528->10.242.241.18:8301: i/o timeout
2021/04/02 00:44:36 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:45:57 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:47:27 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:47:38 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:47:38 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:47:38 [WARN] raft: Rejecting vote request from 10.242.241.18:8300 since we have a leader: 10.242.51.203:8300
2021/04/02 00:47:38 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:47:38 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:47:38 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/04/02 00:47:46 [INFO] consul: New leader elected: dev-consul-server-2
2021/04/02 00:47:47 [DEBUG] raft-net: 10.242.189.8:8300 accepted connection from: 10.242.51.203:38576
2021/04/02 00:50:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:52606->10.242.241.18:8301: i/o timeout
2021/04/02 00:50:18 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:50:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:52626->10.242.241.18:8301: i/o timeout
2021/04/02 00:50:21 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:50:22 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:50:22 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:50:22 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:50:27 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:50:27 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:50:27 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/02 00:50:28 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/04/02 00:50:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:52962->10.242.241.18:8301: i/o timeout
2021/04/02 00:50:47 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:50:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:52974->10.242.241.18:8301: i/o timeout
2021/04/02 00:50:50 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:50:51 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:50:51 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:50:51 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:51:02 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/04/02 00:51:03 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:51:03 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:51:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:53170->10.242.241.18:8301: i/o timeout
2021/04/02 00:51:04 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:51:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:53190->10.242.241.18:8301: i/o timeout
2021/04/02 00:51:07 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:51:08 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:51:08 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:51:08 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:51:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:40190->10.242.241.18:8302: i/o timeout
2021/04/02 00:51:12 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:51:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:40272->10.242.241.18:8302: i/o timeout
2021/04/02 00:51:22 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:51:37 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/04/02 00:51:39 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/04/02 00:51:42 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/02 00:51:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:40566->10.242.241.18:8302: i/o timeout
2021/04/02 00:51:42 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:51:42 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:51:42 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:51:52 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:51:52 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/02 00:51:52 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:51:52 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:51:52 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:51:52 [WARN] raft: Rejecting vote request from 10.242.241.18:8300 since we have a leader: 10.242.51.203:8300
2021/04/02 00:52:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:53942->10.242.241.18:8301: i/o timeout
2021/04/02 00:52:08 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:52:08 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/02 00:54:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:55524->10.242.241.18:8301: i/o timeout
2021/04/02 00:54:20 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:54:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:55572->10.242.241.18:8301: i/o timeout
2021/04/02 00:54:23 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:54:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:55628->10.242.241.18:8301: i/o timeout
2021/04/02 00:54:28 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:54:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:55708->10.242.241.18:8301: i/o timeout
2021/04/02 00:54:33 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:54:35 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:54:35 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:54:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:55722->10.242.241.18:8301: i/o timeout
2021/04/02 00:54:35 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:54:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:42740->10.242.241.18:8302: i/o timeout
2021/04/02 00:54:42 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:54:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:42854->10.242.241.18:8302: i/o timeout
2021/04/02 00:54:52 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:55:06 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/02 00:55:06 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:55:06 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:55:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:43036->10.242.241.18:8302: i/o timeout
2021/04/02 00:55:07 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:55:08 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:56:08 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:56:43 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:56:43 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:56:43 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:56:43 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:56:43 [WARN] raft: Rejecting vote request from 10.242.241.18:8300 since we have a leader: 10.242.51.203:8300
2021/04/02 00:56:43 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/02 01:01:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:60840->10.242.241.18:8301: i/o timeout
2021/04/02 01:01:45 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 01:02:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:32794->10.242.241.18:8301: i/o timeout
2021/04/02 01:02:00 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 01:02:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:33002->10.242.241.18:8301: i/o timeout
2021/04/02 01:02:16 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 01:02:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:33030->10.242.241.18:8301: i/o timeout
2021/04/02 01:02:19 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 01:02:20 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 01:02:20 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 01:02:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:48232->10.242.241.18:8302: i/o timeout
2021/04/02 01:02:22 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 01:02:29 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 01:02:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:48364->10.242.241.18:8302: i/o timeout
2021/04/02 01:02:32 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 01:02:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.189.8:48484->10.242.241.18:8302: i/o timeout
2021/04/02 01:02:42 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 01:02:44 [ERR] memberlist: Push/Pull with dev-consul-server-1.dc1 failed: read tcp 10.242.189.8:48166->10.242.241.18:8302: i/o timeout
2021/04/02 01:02:51 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/02 01:02:51 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 01:02:51 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"