By type
+ CONSUL_BIND=
+ [ -n ]
+ CONSUL_CLIENT=
+ [ -n ]
+ CONSUL_DATA_DIR=/consul/data
+ CONSUL_CONFIG_DIR=/consul/config
+ [ -n ]
+ [ a = - ]
+ [ agent = agent ]
+ shift
+ set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
+ [ consul = consul ]
+ stat -c %u /consul/data
+ id -u consul
+ [ 100 != 100 ]
+ stat -c %u /consul/config
+ id -u consul
+ [ 100 != 100 ]
+ [ ! -z ]
+ exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
bootstrap_expect > 0: expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
Version: 'v1.0.6'
Node ID: 'e3d1abd7-c4ef-d7bc-57e1-b271f273a71b'
Node name: 'dev-consul-server-0'
Datacenter: 'dc1' (Segment: '')
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.242.253.198 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2021/04/17 10:39:17 [INFO] raft: Initial configuration (index=0): []
2021/04/17 10:39:17 [INFO] raft: Node at 10.242.253.198:8300 [Follower] entering Follower state (Leader: "")
2021/04/17 10:39:17 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.253.198
2021/04/17 10:39:17 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.253.198
2021/04/17 10:39:17 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.253.198:8300) (DC: dc1)
2021/04/17 10:39:17 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/04/17 10:39:17 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2021/04/17 10:39:17 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2021/04/17 10:39:17 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
2021/04/17 10:39:17 [INFO] agent: started state syncer
2021/04/17 10:39:17 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer
2021/04/17 10:39:17 [INFO] agent: Joining LAN cluster...
2021/04/17 10:39:17 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/04/17 10:39:17 [WARN] memberlist: Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
2021/04/17 10:39:17 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
2021/04/17 10:39:17 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/04/17 10:39:17 [INFO] agent: (LAN) joined: 0 Err: 3 error(s) occurred:
* Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/04/17 10:39:17 [WARN] agent: Join LAN failed: , retrying in 30s
2021/04/17 10:39:24 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/17 10:39:24 [WARN] raft: no known peers, aborting election
==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=d7699464-6759-9d31-858e-5d4096f0f9e5&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2021/04/17 10:39:47 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.146.73
2021/04/17 10:39:47 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-zlv6m 10.242.253.199
2021/04/17 10:39:47 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.146.73:8300) (DC: dc1)
2021/04/17 10:39:47 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.146.73
2021/04/17 10:39:47 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/17 10:39:47 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/04/17 10:39:47 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/04/17 10:39:47 [INFO] agent: (LAN) joined: 2 Err:
2021/04/17 10:39:47 [INFO] agent: Join LAN completed. Synced with 2 initial agents
2021/04/17 10:39:49 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 10:39:49 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 10:39:50 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.242.244.80:8300,10.242.253.198:8300,10.242.146.73:8300
2021/04/17 10:39:50 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 10:39:50 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 10:39:50 [ERR] agent: Coordinate update error: No cluster leader
2021/04/17 10:39:54 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/17 10:39:54 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/17 10:39:54 [INFO] raft: Node at 10.242.253.198:8300 [Candidate] entering Candidate state in term 2
2021/04/17 10:39:54 [INFO] raft: Election won. Tally: 2
2021/04/17 10:39:54 [INFO] raft: Node at 10.242.253.198:8300 [Leader] entering Leader state
2021/04/17 10:39:54 [INFO] raft: Added peer 79007099-56a4-82c8-2a6b-8367b805feea, starting replication
2021/04/17 10:39:54 [INFO] consul: cluster leadership acquired
2021/04/17 10:39:54 [INFO] raft: Added peer 38d8f917-7a55-e6fb-c4e2-f154d8409d0c, starting replication
2021/04/17 10:39:54 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/17 10:39:54 [INFO] raft: pipelining replication to peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}
2021/04/17 10:39:54 [WARN] raft: AppendEntries to {Voter 38d8f917-7a55-e6fb-c4e2-f154d8409d0c 10.242.146.73:8300} rejected, sending older logs (next: 1)
2021/04/17 10:39:54 [INFO] raft: pipelining replication to peer {Voter 38d8f917-7a55-e6fb-c4e2-f154d8409d0c 10.242.146.73:8300}
2021/04/17 10:39:54 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/04/17 10:39:54 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/17 10:39:54 [INFO] consul: member 'dev-consul-7467bf7b46-zlv6m' joined, marking health alive
2021/04/17 10:39:54 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 10:39:56 [INFO] agent: Synced node info
2021/04/17 11:00:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:51864->10.242.244.80:8301: i/o timeout
2021/04/17 11:00:07 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:00:07 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:00:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:00:11 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:00:15 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500146683s
2021/04/17 11:00:15 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:00:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:00:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52054->10.242.244.80:8301: i/o timeout
2021/04/17 11:00:17 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:00:18 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:00:18 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:00:18 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:00:18 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:00:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:00:19 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:00:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52104->10.242.244.80:8301: i/o timeout
2021/04/17 11:00:20 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:00:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:00:21 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:00:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:37354->10.242.244.80:8302: i/o timeout
2021/04/17 11:00:22 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:00:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:00:23 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500303874s
2021/04/17 11:00:23 [INFO] raft: aborting pipeline replication to peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}
2021/04/17 11:00:23 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:00:23 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:00:23 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:00:23 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:00:24 [INFO] raft: pipelining replication to peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}
2021/04/17 11:00:25 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:00:29 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:00:30 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/17 11:00:31 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:00:31 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 11:00:31 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:00:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:00:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52344->10.242.244.80:8301: i/o timeout
2021/04/17 11:00:34 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:00:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:00:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:00:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52372->10.242.244.80:8301: i/o timeout
2021/04/17 11:00:37 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:00:38 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:00:38 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:00:38 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:00:38 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:00:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:00:38 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:00:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52444->10.242.244.80:8301: i/o timeout
2021/04/17 11:00:39 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:00:39 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:00:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:00:41 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:00:41 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:00:42 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:00:42 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:00:42 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:00:43 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:00:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:00:45 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500444573s
2021/04/17 11:00:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52534->10.242.244.80:8301: i/o timeout
2021/04/17 11:00:46 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:00:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:00:47 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.943482473s
2021/04/17 11:00:48 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:00:48 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:00:48 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:00:48 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:00:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:00:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52614->10.242.244.80:8301: i/o timeout
2021/04/17 11:00:49 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:00:50 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 7.367306923s
2021/04/17 11:00:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:00:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:00:53 [INFO] raft: aborting pipeline replication to peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}
2021/04/17 11:00:53 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:00:53 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:58252->10.242.244.80:8300: i/o timeout
2021/04/17 11:00:53 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:00:53 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:00:53 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:00:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:00:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52692->10.242.244.80:8301: i/o timeout
2021/04/17 11:00:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:00:55 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.51713928s
2021/04/17 11:00:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:00:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:37936->10.242.244.80:8302: i/o timeout
2021/04/17 11:00:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:00:58 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.935967963s
2021/04/17 11:00:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:00:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52790->10.242.244.80:8301: i/o timeout
2021/04/17 11:00:59 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:00:59 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:00:59 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:00:59 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:00:59 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:01:00 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 7.401526979s
2021/04/17 11:01:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:02 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:01:02 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:01:02 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:01:02 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:01:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:03 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.253.198:47554->10.242.244.80:8300: i/o timeout
2021/04/17 11:01:04 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500220108s
2021/04/17 11:01:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52892->10.242.244.80:8301: i/o timeout
2021/04/17 11:01:06 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:01:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:06 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.922032713s
2021/04/17 11:01:08 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:01:08 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:01:08 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:01:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52962->10.242.244.80:8301: i/o timeout
2021/04/17 11:01:09 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:01:09 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 7.407513858s
2021/04/17 11:01:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:12 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:47564->10.242.244.80:8300: i/o timeout
2021/04/17 11:01:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:13 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.253.198:47742->10.242.244.80:8300: i/o timeout
2021/04/17 11:01:13 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:01:13 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:01:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:38222->10.242.244.80:8302: i/o timeout
2021/04/17 11:01:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:01:17 [DEBUG] raft-net: 10.242.253.198:8300 accepted connection from: 10.242.244.80:58314
2021/04/17 11:01:17 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 11:01:18 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:01:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:20 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500413964s
2021/04/17 11:01:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:22 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.976062784s
2021/04/17 11:01:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:23 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.253.198:47914->10.242.244.80:8300: i/o timeout
2021/04/17 11:01:23 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:01:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:24 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 7.45206983s
2021/04/17 11:01:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:38464->10.242.244.80:8302: i/o timeout
2021/04/17 11:01:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:01:28 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:47930->10.242.244.80:8300: i/o timeout
2021/04/17 11:01:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:29 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:01:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:33 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.253.198:48080->10.242.244.80:8300: i/o timeout
2021/04/17 11:01:33 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:01:33 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:01:33 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:01:33 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:01:33 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:41258
2021/04/17 11:01:35 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:01:35 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 11:01:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:38 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:01:38 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:01:38 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:01:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:39 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:48204->10.242.244.80:8300: i/o timeout
2021/04/17 11:01:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:53494->10.242.244.80:8301: i/o timeout
2021/04/17 11:01:40 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:01:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:53542->10.242.244.80:8301: i/o timeout
2021/04/17 11:01:43 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:01:43 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.253.198:48266->10.242.244.80:8300: i/o timeout
2021/04/17 11:01:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:53558->10.242.244.80:8301: i/o timeout
2021/04/17 11:01:46 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:01:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:47 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:01:47 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:01:47 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:01:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:49 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:48382->10.242.244.80:8300: i/o timeout
2021/04/17 11:01:50 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:01:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:38918->10.242.244.80:8302: i/o timeout
2021/04/17 11:01:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:01:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:53 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.253.198:48440->10.242.244.80:8300: i/o timeout
2021/04/17 11:01:53 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:01:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:01:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:00 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:48574->10.242.244.80:8300: i/o timeout
2021/04/17 11:02:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:01 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:02:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:39084->10.242.244.80:8302: i/o timeout
2021/04/17 11:02:02 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:02:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:03 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.253.198:48618->10.242.244.80:8300: i/o timeout
2021/04/17 11:02:03 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:02:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:11 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:48758->10.242.244.80:8300: i/o timeout
2021/04/17 11:02:11 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:02:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:13 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.253.198:48798->10.242.244.80:8300: i/o timeout
2021/04/17 11:02:14 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:02:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:14 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:02:14 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:02:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:39332->10.242.244.80:8302: i/o timeout
2021/04/17 11:02:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:02:17 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:02:18 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:02:18 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:58578
2021/04/17 11:02:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:21 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:48946->10.242.244.80:8300: i/o timeout
2021/04/17 11:02:22 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:02:22 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:02:22 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:02:22 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:02:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:24 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.253.198:48974->10.242.244.80:8300: i/o timeout
2021/04/17 11:02:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:25 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:02:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:39440->10.242.244.80:8302: i/o timeout
2021/04/17 11:02:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:02:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:32 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:49124->10.242.244.80:8300: i/o timeout
2021/04/17 11:02:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:33 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:02:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:35 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.253.198:49184->10.242.244.80:8300: i/o timeout
2021/04/17 11:02:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:38 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:02:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:40 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:02:40 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:02:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:40 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:02:40 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:02:40 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:02:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:43 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:49316->10.242.244.80:8300: i/o timeout
2021/04/17 11:02:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:54602->10.242.244.80:8301: i/o timeout
2021/04/17 11:02:45 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:02:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:47 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:02:47 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:02:47 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:02:47 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:02:48 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:02:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:54674->10.242.244.80:8301: i/o timeout
2021/04/17 11:02:48 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:02:48 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.253.198:49416->10.242.244.80:8300: i/o timeout
2021/04/17 11:02:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:49 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:02:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:50 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:02:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:39954->10.242.244.80:8302: i/o timeout
2021/04/17 11:02:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:02:52 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500305863s
2021/04/17 11:02:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:53 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:02:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:54 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.995684577s
2021/04/17 11:02:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:57 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:02:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:02:59 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.502560571s
2021/04/17 11:03:00 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 11:03:00 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:03:01 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:03:01 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:03:02 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/17 11:03:02 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:03:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:03:03 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:03:03 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.253.198:49656->10.242.244.80:8300: i/o timeout
2021/04/17 11:03:03 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:03:03 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:03:03 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:03:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:03:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:54950->10.242.244.80:8301: i/o timeout
2021/04/17 11:03:05 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:03:05 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500318351s
2021/04/17 11:03:09 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:03:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:55066->10.242.244.80:8301: i/o timeout
2021/04/17 11:03:11 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:03:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:03:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:55130->10.242.244.80:8301: i/o timeout
2021/04/17 11:03:14 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:03:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:03:15 [ERR] raft: peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300} has newer term, stopping replication
2021/04/17 11:03:15 [INFO] raft: Node at 10.242.253.198:8300 [Follower] entering Follower state (Leader: "")
2021/04/17 11:03:15 [INFO] raft: aborting pipeline replication to peer {Voter 38d8f917-7a55-e6fb-c4e2-f154d8409d0c 10.242.146.73:8300}
2021/04/17 11:03:15 [INFO] consul: cluster leadership lost
2021/04/17 11:03:16 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:03:16 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:03:16 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:03:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:40376->10.242.244.80:8302: i/o timeout
2021/04/17 11:03:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:03:20 [DEBUG] raft-net: 10.242.253.198:8300 accepted connection from: 10.242.146.73:48604
2021/04/17 11:03:20 [INFO] consul: New leader elected: dev-consul-server-1
2021/04/17 11:03:21 [DEBUG] raft-net: 10.242.253.198:8300 accepted connection from: 10.242.146.73:48626
2021/04/17 11:03:24 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:50022->10.242.244.80:8300: i/o timeout
2021/04/17 11:03:25 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.146.73:8300
2021/04/17 11:03:27 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/17 11:03:28 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:03:29 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.146.73:8300
2021/04/17 11:03:29 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:03:29 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:03:29 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:03:36 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since our last term is greater (3, 2)
2021/04/17 11:03:40 [INFO] consul: New leader elected: dev-consul-server-1
2021/04/17 11:03:40 [DEBUG] raft-net: 10.242.253.198:8300 accepted connection from: 10.242.146.73:48880
2021/04/17 11:04:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:56316->10.242.244.80:8301: i/o timeout
2021/04/17 11:04:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:04:24 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:04:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:56554->10.242.244.80:8301: i/o timeout
2021/04/17 11:04:32 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:04:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:57008->10.242.244.80:8301: i/o timeout
2021/04/17 11:04:58 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:05:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:57086->10.242.244.80:8301: i/o timeout
2021/04/17 11:05:03 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:05:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:57146->10.242.244.80:8301: i/o timeout
2021/04/17 11:05:07 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:05:07 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:05:07 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:05:07 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:05:07 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:05:07 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:05:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:57220->10.242.244.80:8301: i/o timeout
2021/04/17 11:05:10 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:05:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:42470->10.242.244.80:8302: i/o timeout
2021/04/17 11:05:12 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:05:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:57256->10.242.244.80:8301: i/o timeout
2021/04/17 11:05:12 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:05:13 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.146.73:8300
2021/04/17 11:05:14 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:05:14 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:05:14 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:05:17 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:05:17 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:05:17 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:05:22 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.146.73:8300
2021/04/17 11:05:22 [INFO] consul: New leader elected: dev-consul-server-1
2021/04/17 11:05:23 [DEBUG] raft-net: 10.242.253.198:8300 accepted connection from: 10.242.146.73:50194
2021/04/17 11:05:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:57576->10.242.244.80:8301: i/o timeout
2021/04/17 11:05:31 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:05:32 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:05:32 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:05:32 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:05:32 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:05:32 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:05:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:57804->10.242.244.80:8301: i/o timeout
2021/04/17 11:05:44 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:05:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:57860->10.242.244.80:8301: i/o timeout
2021/04/17 11:05:47 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:05:48 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:05:48 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:05:48 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:05:49 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:05:49 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:05:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:57940->10.242.244.80:8301: i/o timeout
2021/04/17 11:05:52 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:05:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:58038->10.242.244.80:8301: i/o timeout
2021/04/17 11:05:57 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:05:59 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/17 11:06:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:58286->10.242.244.80:8301: i/o timeout
2021/04/17 11:06:11 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:06:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:58336->10.242.244.80:8301: i/o timeout
2021/04/17 11:06:14 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:06:15 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:06:15 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:06:15 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:06:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:43586->10.242.244.80:8302: i/o timeout
2021/04/17 11:06:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:06:17 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:06:20 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:06:20 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:06:20 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:06:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:58516->10.242.244.80:8301: i/o timeout
2021/04/17 11:06:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:58734->10.242.244.80:8301: i/o timeout
2021/04/17 11:06:37 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:06:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:58910->10.242.244.80:8301: i/o timeout
2021/04/17 11:06:47 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:06:50 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/17 11:06:50 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:06:50 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:06:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:58986->10.242.244.80:8301: i/o timeout
2021/04/17 11:06:51 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:06:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:44288->10.242.244.80:8302: i/o timeout
2021/04/17 11:06:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:07:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:44390->10.242.244.80:8302: i/o timeout
2021/04/17 11:07:07 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:07:20 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:07:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:44722->10.242.244.80:8302: i/o timeout
2021/04/17 11:07:22 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:07:22 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:07:22 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:07:22 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:07:47 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:07:54 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:07:54 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:07:54 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:07:54 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:07:54 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:08:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:60536->10.242.244.80:8301: i/o timeout
2021/04/17 11:08:21 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:08:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:60572->10.242.244.80:8301: i/o timeout
2021/04/17 11:08:24 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:08:25 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:08:25 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:08:25 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:08:29 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:08:29 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:08:30 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:08:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:60716->10.242.244.80:8301: i/o timeout
2021/04/17 11:08:31 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:08:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:60754->10.242.244.80:8301: i/o timeout
2021/04/17 11:08:34 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:08:35 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:08:35 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:08:35 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:08:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:46120->10.242.244.80:8302: i/o timeout
2021/04/17 11:08:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:08:45 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:08:45 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:08:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:32852->10.242.244.80:8301: i/o timeout
2021/04/17 11:08:53 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:08:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:32872->10.242.244.80:8301: i/o timeout
2021/04/17 11:08:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:08:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:46366->10.242.244.80:8302: i/o timeout
2021/04/17 11:08:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:08:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:32964->10.242.244.80:8301: i/o timeout
2021/04/17 11:08:58 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:09:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:33010->10.242.244.80:8301: i/o timeout
2021/04/17 11:09:01 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:09:02 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:09:02 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:09:02 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:09:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:46640->10.242.244.80:8302: i/o timeout
2021/04/17 11:09:12 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:09:16 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:09:16 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:09:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:33328->10.242.244.80:8301: i/o timeout
2021/04/17 11:09:19 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:09:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:33352->10.242.244.80:8301: i/o timeout
2021/04/17 11:09:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:09:23 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:09:23 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:09:23 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:09:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:46890->10.242.244.80:8302: i/o timeout
2021/04/17 11:09:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:09:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:47062->10.242.244.80:8302: i/o timeout
2021/04/17 11:09:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:09:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:47226->10.242.244.80:8302: i/o timeout
2021/04/17 11:09:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:09:47 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:09:47 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:09:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:33868->10.242.244.80:8301: i/o timeout
2021/04/17 11:09:51 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:09:53 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:09:53 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:09:57 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:09:57 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:09:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:47332->10.242.244.80:8302: i/o timeout
2021/04/17 11:09:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:09:57 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:10:00 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:10:08 [ERR] memberlist: failed to receive: read tcp 10.242.253.198:8301->10.242.244.80:40984: i/o timeout from=10.242.244.80:40984
2021/04/17 11:10:08 [ERR] memberlist: Failed to send error: write tcp 10.242.253.198:8301->10.242.244.80:40984: i/o timeout from=10.242.244.80:40984
2021/04/17 11:10:10 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:10:10 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:10:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:47736->10.242.244.80:8302: i/o timeout
2021/04/17 11:10:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:10:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:47910->10.242.244.80:8302: i/o timeout
2021/04/17 11:10:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:10:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:48082->10.242.244.80:8302: i/o timeout
2021/04/17 11:10:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:10:37 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:10:40 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:10:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:48250->10.242.244.80:8302: i/o timeout
2021/04/17 11:10:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:10:47 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:10:47 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:10:47 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:10:49 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:10:49 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:10:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:34908->10.242.244.80:8301: i/o timeout
2021/04/17 11:10:52 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:10:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:34930->10.242.244.80:8301: i/o timeout
2021/04/17 11:10:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:10:55 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:10:55 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:10:55 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:11:20 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:11:21 [ERR] memberlist: failed to receive: read tcp 10.242.253.198:8302->10.242.244.80:34060: i/o timeout from=10.242.244.80:34060
2021/04/17 11:11:21 [ERR] memberlist: Failed to send error: write tcp 10.242.253.198:8302->10.242.244.80:34060: i/o timeout from=10.242.244.80:34060
2021/04/17 11:12:24 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:13:00 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:13:09 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:13:09 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:13:09 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:13:10 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:13:10 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:13:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:37328->10.242.244.80:8301: i/o timeout
2021/04/17 11:13:11 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:13:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:37358->10.242.244.80:8301: i/o timeout
2021/04/17 11:13:14 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:13:15 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:13:15 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:13:15 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:13:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:51038->10.242.244.80:8302: i/o timeout
2021/04/17 11:13:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:13:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:51208->10.242.244.80:8302: i/o timeout
2021/04/17 11:13:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:13:47 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:13:47 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:13:47 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:13:49 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:13:49 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:38586
2021/04/17 11:13:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:51476->10.242.244.80:8302: i/o timeout
2021/04/17 11:13:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:14:24 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:14:48 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:14:48 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:14:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:39026->10.242.244.80:8301: i/o timeout
2021/04/17 11:14:51 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:14:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:39084->10.242.244.80:8301: i/o timeout
2021/04/17 11:14:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:14:55 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:14:55 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:14:55 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:15:18 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:15:19 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:15:19 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:15:19 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:15:19 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:15:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:39536->10.242.244.80:8301: i/o timeout
2021/04/17 11:15:21 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:15:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:39594->10.242.244.80:8301: i/o timeout
2021/04/17 11:15:25 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:15:25 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:15:25 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:15:25 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:15:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:39616->10.242.244.80:8301: i/o timeout
2021/04/17 11:15:28 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:15:32 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/17 11:15:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:53350->10.242.244.80:8302: i/o timeout
2021/04/17 11:15:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:15:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:53518->10.242.244.80:8302: i/o timeout
2021/04/17 11:15:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:16:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:53750->10.242.244.80:8302: i/o timeout
2021/04/17 11:16:07 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:16:12 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:16:12 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:16:12 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:16:12 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:16:17 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:16:17 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:16:17 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:16:17 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:18:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:42304->10.242.244.80:8301: i/o timeout
2021/04/17 11:18:05 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:18:06 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:18:06 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:18:06 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:18:06 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:18:06 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:18:11 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/17 11:18:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:42528->10.242.244.80:8301: i/o timeout
2021/04/17 11:18:17 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:18:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:42584->10.242.244.80:8301: i/o timeout
2021/04/17 11:18:20 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:18:21 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:18:21 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:18:21 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:18:27 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/17 11:18:29 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:18:29 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:18:29 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:18:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:42772->10.242.244.80:8301: i/o timeout
2021/04/17 11:18:32 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:18:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:42824->10.242.244.80:8301: i/o timeout
2021/04/17 11:18:34 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:18:36 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:18:36 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:18:36 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:18:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:56408->10.242.244.80:8302: i/o timeout
2021/04/17 11:18:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:18:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:56640->10.242.244.80:8302: i/o timeout
2021/04/17 11:18:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:19:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:56798->10.242.244.80:8302: i/o timeout
2021/04/17 11:19:12 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:19:12 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:19:12 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:19:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:56980->10.242.244.80:8302: i/o timeout
2021/04/17 11:19:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:19:19 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:20:20 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:20:20 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:20:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:58154->10.242.244.80:8302: i/o timeout
2021/04/17 11:20:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:20:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:58334->10.242.244.80:8302: i/o timeout
2021/04/17 11:20:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:20:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:58508->10.242.244.80:8302: i/o timeout
2021/04/17 11:20:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:20:57 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:20:57 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:20:57 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:21:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:58782->10.242.244.80:8302: i/o timeout
2021/04/17 11:21:02 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:21:15 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:21:15 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:21:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:59112->10.242.244.80:8302: i/o timeout
2021/04/17 11:21:22 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:21:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:59174->10.242.244.80:8302: i/o timeout
2021/04/17 11:21:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:21:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:59444->10.242.244.80:8302: i/o timeout
2021/04/17 11:21:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:21:52 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:21:52 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:21:52 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:21:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:59686->10.242.244.80:8302: i/o timeout
2021/04/17 11:21:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:22:39 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:23:19 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:23:56 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:23:56 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:24:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:33752->10.242.244.80:8302: i/o timeout
2021/04/17 11:24:12 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:24:19 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:24:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:33986->10.242.244.80:8302: i/o timeout
2021/04/17 11:24:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:24:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:34100->10.242.244.80:8302: i/o timeout
2021/04/17 11:24:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:24:42 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:24:42 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:24:42 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:24:43 [ERR] memberlist: failed to receive: read tcp 10.242.253.198:8302->10.242.244.80:42912: i/o timeout from=10.242.244.80:42912
2021/04/17 11:24:43 [ERR] memberlist: Failed to send error: write tcp 10.242.253.198:8302->10.242.244.80:42912: i/o timeout from=10.242.244.80:42912
2021/04/17 11:24:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:34368->10.242.244.80:8302: i/o timeout
2021/04/17 11:24:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:24:49 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:25:05 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:25:05 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:25:05 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/04/17 11:25:10 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:25:10 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:25:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:49632->10.242.244.80:8301: i/o timeout
2021/04/17 11:25:11 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:25:15 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:25:15 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:25:15 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:25:15 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/17 11:25:15 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.146.73:8300
2021/04/17 11:25:17 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/17 11:25:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:35144->10.242.244.80:8302: i/o timeout
2021/04/17 11:25:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:25:31 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:25:31 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:25:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:50056->10.242.244.80:8301: i/o timeout
2021/04/17 11:25:33 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:25:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:50080->10.242.244.80:8301: i/o timeout
2021/04/17 11:25:36 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:25:37 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:25:37 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:25:37 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:25:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:35332->10.242.244.80:8302: i/o timeout
2021/04/17 11:25:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:25:42 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:25:42 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:25:59 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:25:59 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:26:00 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:26:00 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:26:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:50966->10.242.244.80:8301: i/o timeout
2021/04/17 11:26:24 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:27:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:51804->10.242.244.80:8301: i/o timeout
2021/04/17 11:27:06 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:27:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:51902->10.242.244.80:8301: i/o timeout
2021/04/17 11:27:10 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:27:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:51940->10.242.244.80:8301: i/o timeout
2021/04/17 11:27:13 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:27:14 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:27:14 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:27:14 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:27:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:51986->10.242.244.80:8301: i/o timeout
2021/04/17 11:27:15 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:27:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:37402->10.242.244.80:8302: i/o timeout
2021/04/17 11:27:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:27:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:37516->10.242.244.80:8302: i/o timeout
2021/04/17 11:27:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:27:47 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:27:47 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:27:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:37882->10.242.244.80:8302: i/o timeout
2021/04/17 11:27:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:28:03 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:28:03 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:28:05 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:28:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52894->10.242.244.80:8301: i/o timeout
2021/04/17 11:28:05 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:28:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52998->10.242.244.80:8301: i/o timeout
2021/04/17 11:28:09 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:28:09 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:28:09 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:28:09 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:28:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:53016->10.242.244.80:8301: i/o timeout
2021/04/17 11:28:12 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:28:19 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:28:19 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:28:19 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:28:24 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.146.73:8300
2021/04/17 11:28:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:38530->10.242.244.80:8302: i/o timeout
2021/04/17 11:28:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:28:43 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:28:43 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:49460
2021/04/17 11:28:47 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:28:47 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:28:47 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/17 11:28:47 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.146.73:8300
2021/04/17 11:28:49 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/17 11:28:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:53790->10.242.244.80:8301: i/o timeout
2021/04/17 11:28:50 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:28:53 [WARN] raft: Heartbeat timeout from "10.242.146.73:8300" reached, starting election
2021/04/17 11:28:53 [INFO] raft: Node at 10.242.253.198:8300 [Candidate] entering Candidate state in term 12
2021/04/17 11:28:53 [INFO] raft: Node at 10.242.253.198:8300 [Follower] entering Follower state (Leader: "")
2021/04/17 11:28:53 [INFO] consul: New leader elected: dev-consul-server-1
2021/04/17 11:28:54 [DEBUG] raft-net: 10.242.253.198:8300 accepted connection from: 10.242.146.73:42580
2021/04/17 11:29:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:54064->10.242.244.80:8301: i/o timeout
2021/04/17 11:29:06 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:29:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:54292->10.242.244.80:8301: i/o timeout
2021/04/17 11:29:17 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:29:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:54354->10.242.244.80:8301: i/o timeout
2021/04/17 11:29:20 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:29:20 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:30:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:55524->10.242.244.80:8301: i/o timeout
2021/04/17 11:30:25 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:30:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:55574->10.242.244.80:8301: i/o timeout
2021/04/17 11:30:28 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:30:29 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:30:29 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:30:29 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:30:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:40960->10.242.244.80:8302: i/o timeout
2021/04/17 11:30:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:30:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:41078->10.242.244.80:8302: i/o timeout
2021/04/17 11:30:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:30:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:41344->10.242.244.80:8302: i/o timeout
2021/04/17 11:30:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:31:00 [ERR] memberlist: Push/Pull with dev-consul-server-2.dc1 failed: read tcp 10.242.253.198:40894->10.242.244.80:8302: i/o timeout
2021/04/17 11:31:07 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:31:07 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:31:07 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:31:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:41634->10.242.244.80:8302: i/o timeout
2021/04/17 11:31:12 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:31:17 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.146.73:8300
2021/04/17 11:31:17 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:31:17 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:31:19 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:31:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:56600->10.242.244.80:8301: i/o timeout
2021/04/17 11:31:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:31:24 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:31:24 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:31:24 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:31:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:56658->10.242.244.80:8301: i/o timeout
2021/04/17 11:31:25 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:31:39 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:31:50 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.146.73:8300
2021/04/17 11:31:51 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:31:51 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:31:51 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:31:51 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:31:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:57272->10.242.244.80:8301: i/o timeout
2021/04/17 11:31:57 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:32:00 [INFO] consul: New leader elected: dev-consul-server-1
2021/04/17 11:32:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:57352->10.242.244.80:8301: i/o timeout
2021/04/17 11:32:01 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:32:01 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:32:01 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:32:01 [DEBUG] raft-net: 10.242.253.198:8300 accepted connection from: 10.242.146.73:45420
2021/04/17 11:32:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:57370->10.242.244.80:8301: i/o timeout
2021/04/17 11:32:04 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:32:09 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:32:09 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:32:12 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.146.73:8300
2021/04/17 11:32:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:57604->10.242.244.80:8301: i/o timeout
2021/04/17 11:32:14 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:32:15 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:32:15 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:32:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:42860->10.242.244.80:8302: i/o timeout
2021/04/17 11:32:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:32:17 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:32:27 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:32:27 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:51912
2021/04/17 11:32:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:43178->10.242.244.80:8302: i/o timeout
2021/04/17 11:32:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:32:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:43250->10.242.244.80:8302: i/o timeout
2021/04/17 11:32:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:32:47 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:32:47 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:32:47 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:32:55 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:32:57 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:33:49 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:34:16 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:34:16 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:34:16 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:34:19 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.146.73:8300
2021/04/17 11:34:23 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:34:23 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:34:28 [WARN] raft: Heartbeat timeout from "10.242.146.73:8300" reached, starting election
2021/04/17 11:34:28 [INFO] raft: Node at 10.242.253.198:8300 [Candidate] entering Candidate state in term 16
2021/04/17 11:34:28 [INFO] raft: Election won. Tally: 2
2021/04/17 11:34:28 [INFO] raft: Node at 10.242.253.198:8300 [Leader] entering Leader state
2021/04/17 11:34:28 [INFO] raft: Added peer 79007099-56a4-82c8-2a6b-8367b805feea, starting replication
2021/04/17 11:34:28 [INFO] raft: Added peer 38d8f917-7a55-e6fb-c4e2-f154d8409d0c, starting replication
2021/04/17 11:34:28 [INFO] consul: cluster leadership acquired
2021/04/17 11:34:28 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/17 11:34:28 [INFO] raft: pipelining replication to peer {Voter 38d8f917-7a55-e6fb-c4e2-f154d8409d0c 10.242.146.73:8300}
2021/04/17 11:34:28 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:34:28 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 11:34:28 [ERR] raft: peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300} has newer term, stopping replication
2021/04/17 11:34:28 [INFO] raft: Node at 10.242.253.198:8300 [Follower] entering Follower state (Leader: "")
2021/04/17 11:34:28 [INFO] consul: cluster leadership lost
2021/04/17 11:34:28 [INFO] raft: aborting pipeline replication to peer {Voter 38d8f917-7a55-e6fb-c4e2-f154d8409d0c 10.242.146.73:8300}
2021/04/17 11:34:30 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:34:30 [WARN] consul.coordinate: Batch update failed: node is not the leader
2021/04/17 11:34:32 [INFO] serf: EventMemberUpdate: dev-consul-server-2.dc1
2021/04/17 11:34:32 [INFO] serf: EventMemberUpdate: dev-consul-server-2
2021/04/17 11:34:34 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/17 11:34:34 [INFO] raft: Node at 10.242.253.198:8300 [Candidate] entering Candidate state in term 17
2021/04/17 11:34:34 [ERR] raft: Failed to make RequestVote RPC to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: EOF
2021/04/17 11:34:34 [INFO] raft: Election won. Tally: 2
2021/04/17 11:34:34 [INFO] raft: Node at 10.242.253.198:8300 [Leader] entering Leader state
2021/04/17 11:34:34 [INFO] raft: Added peer 79007099-56a4-82c8-2a6b-8367b805feea, starting replication
2021/04/17 11:34:34 [INFO] raft: Added peer 38d8f917-7a55-e6fb-c4e2-f154d8409d0c, starting replication
2021/04/17 11:34:34 [INFO] consul: cluster leadership acquired
2021/04/17 11:34:34 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/17 11:34:34 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: EOF
2021/04/17 11:34:34 [INFO] raft: pipelining replication to peer {Voter 38d8f917-7a55-e6fb-c4e2-f154d8409d0c 10.242.146.73:8300}
2021/04/17 11:34:34 [INFO] raft: Updating configuration with RemoveServer (79007099-56a4-82c8-2a6b-8367b805feea, ) to [{Suffrage:Voter ID:e3d1abd7-c4ef-d7bc-57e1-b271f273a71b Address:10.242.253.198:8300} {Suffrage:Voter ID:38d8f917-7a55-e6fb-c4e2-f154d8409d0c Address:10.242.146.73:8300}]
2021/04/17 11:34:34 [INFO] raft: Removed peer 79007099-56a4-82c8-2a6b-8367b805feea, stopping replication after 781
2021/04/17 11:34:34 [INFO] consul: removed server with duplicate address: 10.242.244.80:8300
2021/04/17 11:34:34 [INFO] raft: Updating configuration with AddNonvoter (2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, 10.242.244.80:8300) to [{Suffrage:Voter ID:e3d1abd7-c4ef-d7bc-57e1-b271f273a71b Address:10.242.253.198:8300} {Suffrage:Voter ID:38d8f917-7a55-e6fb-c4e2-f154d8409d0c Address:10.242.146.73:8300} {Suffrage:Nonvoter ID:2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe Address:10.242.244.80:8300}]
2021/04/17 11:34:34 [INFO] raft: Added peer 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, starting replication
2021/04/17 11:34:34 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:34:35 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:34:36 [WARN] raft: AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300} rejected, sending older logs (next: 1)
2021/04/17 11:34:36 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:34:36 [WARN] raft: AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300} rejected, sending older logs (next: 1)
2021/04/17 11:34:37 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:34:37 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:34:38 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:34:39 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:34:39 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 11:34:39 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:34:40 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:34:40 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:34:40 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:34:41 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:34:41 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:34:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:34:43 [WARN] raft: Failed to contact 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe in 2.500305646s
2021/04/17 11:34:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:34:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:60538->10.242.244.80:8301: i/o timeout
2021/04/17 11:34:46 [WARN] raft: Failed to contact 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe in 4.96409635s
2021/04/17 11:34:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:34:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:45766->10.242.244.80:8302: i/o timeout
2021/04/17 11:34:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:60578->10.242.244.80:8301: i/o timeout
2021/04/17 11:34:47 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:34:48 [WARN] raft: Failed to contact 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe in 7.375542322s
2021/04/17 11:34:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:34:50 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: read tcp 10.242.253.198:55214->10.242.244.80:8300: i/o timeout
2021/04/17 11:34:50 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:34:50 [INFO] raft: pipelining replication to peer {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}
2021/04/17 11:34:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:60650->10.242.244.80:8301: i/o timeout
2021/04/17 11:34:50 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:34:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:34:51 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:34:51 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:34:51 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:34:51 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:55224->10.242.244.80:8300: i/o timeout
2021/04/17 11:34:52 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:34:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:34:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:34:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:34:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:45980->10.242.244.80:8302: i/o timeout
2021/04/17 11:34:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:34:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:34:59 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/04/17 11:35:00 [ERR] raft: Failed to pipeline AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: write tcp 10.242.253.198:55236->10.242.244.80:8300: use of closed network connection
2021/04/17 11:35:00 [INFO] raft: aborting pipeline replication to peer {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}
2021/04/17 11:35:00 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:35:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:35:01 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:35:01 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:35:01 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:35:03 [INFO] raft: pipelining replication to peer {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}
2021/04/17 11:35:24 [INFO] autopilot: Promoting Server (ID: "2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe" Address: "10.242.244.80:8300") to voter
2021/04/17 11:35:24 [INFO] raft: Updating configuration with AddStaging (2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, 10.242.244.80:8300) to [{Suffrage:Voter ID:e3d1abd7-c4ef-d7bc-57e1-b271f273a71b Address:10.242.253.198:8300} {Suffrage:Voter ID:38d8f917-7a55-e6fb-c4e2-f154d8409d0c Address:10.242.146.73:8300} {Suffrage:Voter ID:2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe Address:10.242.244.80:8300}]
2021/04/17 11:35:41 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:35:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:35:43 [WARN] raft: Failed to contact 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe in 2.500679497s
2021/04/17 11:35:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:33434->10.242.244.80:8301: i/o timeout
2021/04/17 11:35:44 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:35:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:35:45 [WARN] raft: Failed to contact 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe in 2.500230388s
2021/04/17 11:35:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:35:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:33462->10.242.244.80:8301: i/o timeout
2021/04/17 11:35:47 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:35:47 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:35:47 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:35:47 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:35:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:35:49 [WARN] raft: Failed to contact 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe in 2.505567233s
2021/04/17 11:35:50 [INFO] raft: aborting pipeline replication to peer {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}
2021/04/17 11:35:50 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:35:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:35:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:35:53 [WARN] raft: Failed to contact 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe in 2.503901676s
2021/04/17 11:35:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:35:55 [WARN] raft: Failed to contact 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe in 4.952008248s
2021/04/17 11:35:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:35:57 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:55602->10.242.244.80:8300: i/o timeout
2021/04/17 11:35:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:47114->10.242.244.80:8302: i/o timeout
2021/04/17 11:35:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:35:57 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:35:58 [WARN] raft: Failed to contact 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe in 7.391221493s
2021/04/17 11:35:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:00 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: read tcp 10.242.253.198:56704->10.242.244.80:8300: i/o timeout
2021/04/17 11:36:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:01 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:36:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:47234->10.242.244.80:8302: i/o timeout
2021/04/17 11:36:07 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:36:07 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:56844->10.242.244.80:8300: i/o timeout
2021/04/17 11:36:08 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:36:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:11 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: read tcp 10.242.253.198:56900->10.242.244.80:8300: i/o timeout
2021/04/17 11:36:11 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:36:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:47510->10.242.244.80:8302: i/o timeout
2021/04/17 11:36:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:36:18 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:57042->10.242.244.80:8300: i/o timeout
2021/04/17 11:36:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:19 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:36:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:21 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: read tcp 10.242.253.198:57088->10.242.244.80:8300: i/o timeout
2021/04/17 11:36:21 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:36:22 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:36:22 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:36:22 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:36:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:29 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:57262->10.242.244.80:8300: i/o timeout
2021/04/17 11:36:29 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:36:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:31 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: read tcp 10.242.253.198:57292->10.242.244.80:8300: i/o timeout
2021/04/17 11:36:31 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:36:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:33 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:36:33 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:36:33 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:36:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:34460->10.242.244.80:8301: i/o timeout
2021/04/17 11:36:37 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:36:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:39 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:57462->10.242.244.80:8300: i/o timeout
2021/04/17 11:36:40 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:36:40 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:36:40 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:36:40 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:36:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:41 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: read tcp 10.242.253.198:57484->10.242.244.80:8300: i/o timeout
2021/04/17 11:36:41 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:36:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:34550->10.242.244.80:8301: i/o timeout
2021/04/17 11:36:41 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:36:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:49 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:36:50 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:57668->10.242.244.80:8300: i/o timeout
2021/04/17 11:36:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:51 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: read tcp 10.242.253.198:57682->10.242.244.80:8300: i/o timeout
2021/04/17 11:36:51 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:36:51 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:36:52 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:36:52 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:54364
2021/04/17 11:36:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:36:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:01 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: read tcp 10.242.253.198:57878->10.242.244.80:8300: i/o timeout
2021/04/17 11:37:01 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:57884->10.242.244.80:8300: i/o timeout
2021/04/17 11:37:01 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:37:02 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:37:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:07 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:37:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:11 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: read tcp 10.242.253.198:58076->10.242.244.80:8300: i/o timeout
2021/04/17 11:37:12 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:37:12 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:58094->10.242.244.80:8300: i/o timeout
2021/04/17 11:37:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:13 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:37:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:22 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: read tcp 10.242.253.198:58276->10.242.244.80:8300: i/o timeout
2021/04/17 11:37:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:23 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:37:23 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:58312->10.242.244.80:8300: i/o timeout
2021/04/17 11:37:24 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:37:24 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:37:25 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:37:25 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:37:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:27 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:37:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:33 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: read tcp 10.242.253.198:58488->10.242.244.80:8300: i/o timeout
2021/04/17 11:37:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:35 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:58526->10.242.244.80:8300: i/o timeout
2021/04/17 11:37:36 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:37:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:39 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:37:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:46 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: read tcp 10.242.253.198:58730->10.242.244.80:8300: i/o timeout
2021/04/17 11:37:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:49 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:58802->10.242.244.80:8300: i/o timeout
2021/04/17 11:37:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:51 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:37:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:55 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:37:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:37:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:01 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: read tcp 10.242.253.198:59030->10.242.244.80:8300: i/o timeout
2021/04/17 11:38:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:04 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:38:04 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:38:05 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:59092->10.242.244.80:8300: i/o timeout
2021/04/17 11:38:07 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:38:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:11 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:38:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:16 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:38:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:21 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: read tcp 10.242.253.198:59414->10.242.244.80:8300: i/o timeout
2021/04/17 11:38:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:26 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:59510->10.242.244.80:8300: i/o timeout
2021/04/17 11:38:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:31 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:38:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:37 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:38:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:41 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:38:41 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: read tcp 10.242.253.198:59806->10.242.244.80:8300: i/o timeout
2021/04/17 11:38:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:43 [WARN] raft: Failed to contact 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe in 2.500370755s
2021/04/17 11:38:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:45 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:38:45 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:38:45 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:38:45 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:38:45 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:38:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:36974->10.242.244.80:8301: i/o timeout
2021/04/17 11:38:47 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:38:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:36994->10.242.244.80:8301: i/o timeout
2021/04/17 11:38:50 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:38:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:51 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:38:51 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:38:51 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:38:51 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:38:51 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:38:52 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:38:52 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:38:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:54 [WARN] raft: Failed to contact 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe in 2.500991356s
2021/04/17 11:38:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:56 [WARN] raft: Failed to contact 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe in 4.976493289s
2021/04/17 11:38:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:38:57 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:38:58 [WARN] raft: Failed to contact 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe in 7.432571136s
2021/04/17 11:38:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:39:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:39:02 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: read tcp 10.242.253.198:59918->10.242.244.80:8300: i/o timeout
2021/04/17 11:39:02 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:60204->10.242.244.80:8300: i/o timeout
2021/04/17 11:39:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:39:03 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:39:03 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:39:04 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:39:04 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:39:04 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:39:04 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:39:04 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:39:09 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:39:13 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:39:14 [WARN] raft: Failed to contact 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe in 2.500331695s
2021/04/17 11:39:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:39:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:39:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:37574->10.242.244.80:8301: i/o timeout
2021/04/17 11:39:17 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:39:18 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:39:18 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:39:18 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:39:18 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:39:18 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:39:18 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:39:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:39:19 [INFO] raft: pipelining replication to peer {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}
2021/04/17 11:39:19 [INFO] raft: aborting pipeline replication to peer {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}
2021/04/17 11:39:19 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: read tcp 10.242.253.198:60620->10.242.244.80:8300: read: connection reset by peer
2021/04/17 11:39:19 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:19 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:19 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:19 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:20 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:20 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:20 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:39:20 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:20 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:20 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:20 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: stream closed
2021/04/17 11:39:21 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:21 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:21 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:21 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:39:21 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:39:22 [WARN] raft: Failed to contact 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe in 2.500327056s
2021/04/17 11:39:22 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:22 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:22 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:23 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:39:23 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:39:23 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 11:39:23 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:24 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:39:24 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:39:24 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:39:24 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:39:24 [WARN] raft: Failed to contact 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe in 4.953738913s
2021/04/17 11:39:24 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:25 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:39:25 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:25 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:39:25 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:25 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:39:25 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:39:26 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:39:26 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:26 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:26 [WARN] raft: Failed to contact 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe in 7.416237503s
2021/04/17 11:39:27 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:39:27 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:27 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:39:28 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:29 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:39:29 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:29 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:39:30 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:39:30 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:30 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:31 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:39:32 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:39:32 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:32 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:33 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:39:34 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:35 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:39:35 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:39:36 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:39:37 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:39:38 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:39:38 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:38 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:39 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:39:40 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:39:40 [ERR] raft: Failed to AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300}: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:40 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:41 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:39:41 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 11:39:41 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:39:42 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:43 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:39:44 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:45 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:39:46 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.244.80:8300: getsockopt: connection refused
2021/04/17 11:39:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:39:47 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:39:47 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:39:49 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:39:51 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:39:51 [WARN] raft: AppendEntries to {Nonvoter 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe 10.242.244.80:8300} rejected, sending older logs (next: 1)
2021/04/17 11:39:51 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:39:51 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:39:51 [INFO] serf: EventMemberUpdate: dev-consul-server-2.dc1
2021/04/17 11:39:51 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:39:51 [WARN] Unable to get address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, using fallback address 10.242.244.80:8300: Could not find address for server id 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe
2021/04/17 11:39:51 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:39:51 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:39:51 [INFO] raft: Updating configuration with RemoveServer (2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, ) to [{Suffrage:Voter ID:e3d1abd7-c4ef-d7bc-57e1-b271f273a71b Address:10.242.253.198:8300} {Suffrage:Voter ID:38d8f917-7a55-e6fb-c4e2-f154d8409d0c Address:10.242.146.73:8300}]
2021/04/17 11:39:51 [INFO] raft: Removed peer 2a37ce48-1153-2fc4-9ba6-ad71bc74e0fe, stopping replication after 855
2021/04/17 11:39:51 [INFO] consul: removed server with duplicate address: 10.242.244.80:8300
2021/04/17 11:39:51 [INFO] raft: Updating configuration with AddNonvoter (a06ee723-0a7a-bed8-b418-2a3a48a30906, 10.242.244.80:8300) to [{Suffrage:Voter ID:e3d1abd7-c4ef-d7bc-57e1-b271f273a71b Address:10.242.253.198:8300} {Suffrage:Voter ID:38d8f917-7a55-e6fb-c4e2-f154d8409d0c Address:10.242.146.73:8300} {Suffrage:Nonvoter ID:a06ee723-0a7a-bed8-b418-2a3a48a30906 Address:10.242.244.80:8300}]
2021/04/17 11:39:51 [INFO] raft: Added peer a06ee723-0a7a-bed8-b418-2a3a48a30906, starting replication
2021/04/17 11:39:51 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:39:51 [WARN] raft: AppendEntries to {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300} rejected, sending older logs (next: 257)
2021/04/17 11:39:51 [INFO] raft: pipelining replication to peer {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 11:40:04 [INFO] autopilot: Promoting Server (ID: "a06ee723-0a7a-bed8-b418-2a3a48a30906" Address: "10.242.244.80:8300") to voter
2021/04/17 11:40:04 [INFO] raft: Updating configuration with AddStaging (a06ee723-0a7a-bed8-b418-2a3a48a30906, 10.242.244.80:8300) to [{Suffrage:Voter ID:e3d1abd7-c4ef-d7bc-57e1-b271f273a71b Address:10.242.253.198:8300} {Suffrage:Voter ID:38d8f917-7a55-e6fb-c4e2-f154d8409d0c Address:10.242.146.73:8300} {Suffrage:Voter ID:a06ee723-0a7a-bed8-b418-2a3a48a30906 Address:10.242.244.80:8300}]
2021/04/17 11:40:17 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:40:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:40:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:38838->10.242.244.80:8301: i/o timeout
2021/04/17 11:40:19 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:40:21 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:40:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:38854->10.242.244.80:8301: i/o timeout
2021/04/17 11:40:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:40:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:40:22 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500320101s
2021/04/17 11:40:23 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:40:23 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:40:23 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:40:23 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:40:25 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:40:25 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:40:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:40:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52414->10.242.244.80:8302: i/o timeout
2021/04/17 11:40:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:40:27 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500368287s
2021/04/17 11:40:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:40:30 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.970461498s
2021/04/17 11:40:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:40:32 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.440150778s
2021/04/17 11:40:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:40:33 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:40:33 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:40:33 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:40:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:39116->10.242.244.80:8301: i/o timeout
2021/04/17 11:40:34 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:40:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:40:35 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.50026264s
2021/04/17 11:40:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:40:38 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.928596709s
2021/04/17 11:40:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:39206->10.242.244.80:8301: i/o timeout
2021/04/17 11:40:38 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:40:38 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:40:38 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:40:38 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:40:38 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:40:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:40:40 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.415227951s
2021/04/17 11:40:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:40:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:40:43 [ERR] raft: Failed to pipeline AppendEntries to {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: write tcp 10.242.253.198:33180->10.242.244.80:8300: use of closed network connection
2021/04/17 11:40:43 [INFO] raft: aborting pipeline replication to peer {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 11:40:43 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:40:43 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:33202->10.242.244.80:8300: i/o timeout
2021/04/17 11:40:44 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:40:45 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.501206586s
2021/04/17 11:40:45 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:40:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:40:47 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:40:48 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.969385957s
2021/04/17 11:40:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:40:50 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.457023314s
2021/04/17 11:40:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:40:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52916->10.242.244.80:8302: i/o timeout
2021/04/17 11:40:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:40:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:40:53 [ERR] raft: Failed to AppendEntries to {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:34184->10.242.244.80:8300: i/o timeout
2021/04/17 11:40:53 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:40:54 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:34208->10.242.244.80:8300: i/o timeout
2021/04/17 11:40:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:40:55 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:40:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:40:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:41:00 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:41:00 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:41:01 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:41:01 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 11:41:01 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:41:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:53104->10.242.244.80:8302: i/o timeout
2021/04/17 11:41:02 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:41:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:41:03 [ERR] raft: Failed to AppendEntries to {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:34382->10.242.244.80:8300: i/o timeout
2021/04/17 11:41:03 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:41:04 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:41:04 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:58032
2021/04/17 11:41:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:41:05 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:34416->10.242.244.80:8300: i/o timeout
2021/04/17 11:41:05 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:41:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:41:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:41:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:41:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:53294->10.242.244.80:8302: i/o timeout
2021/04/17 11:41:12 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:41:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:41:13 [ERR] raft: Failed to AppendEntries to {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:34572->10.242.244.80:8300: i/o timeout
2021/04/17 11:41:13 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:41:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:41:15 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:34622->10.242.244.80:8300: i/o timeout
2021/04/17 11:41:16 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:41:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:41:17 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:41:17 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:41:17 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:41:18 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:41:18 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:41:18 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:41:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:41:19 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:41:19 [DEBUG] raft-net: 10.242.253.198:8300 accepted connection from: 10.242.244.80:59172
2021/04/17 11:41:19 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 11:41:19 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:41:19 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:41:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:41:21 [INFO] raft: pipelining replication to peer {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 11:41:25 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:41:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:41:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:40134->10.242.244.80:8301: i/o timeout
2021/04/17 11:41:27 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:41:27 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:41:27 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500303248s
2021/04/17 11:41:29 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:41:29 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:41:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:40274->10.242.244.80:8301: i/o timeout
2021/04/17 11:41:35 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:41:35 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 11:41:35 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:41:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:40306->10.242.244.80:8301: i/o timeout
2021/04/17 11:41:36 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:41:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:41:37 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.501041481s
2021/04/17 11:41:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:41:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:40384->10.242.244.80:8301: i/o timeout
2021/04/17 11:41:39 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:41:39 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:41:39 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:41:39 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:41:39 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:41:40 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:41:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:41:41 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:41:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:53878->10.242.244.80:8302: i/o timeout
2021/04/17 11:41:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:41:42 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:41:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:41:43 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:41:44 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:41:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:41:45 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:41:46 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:41:46 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:41:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:41:48 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500368379s
2021/04/17 11:41:48 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:41:48 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:41:48 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:41:48 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:41:51 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:41:51 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 11:41:55 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:41:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:41:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:40748->10.242.244.80:8301: i/o timeout
2021/04/17 11:41:58 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:41:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:40792->10.242.244.80:8301: i/o timeout
2021/04/17 11:42:01 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:42:01 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:42:01 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:42:01 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:42:01 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:42:02 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500269919s
2021/04/17 11:42:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:04 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.960891308s
2021/04/17 11:42:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:06 [INFO] raft: aborting pipeline replication to peer {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 11:42:06 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:42:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:09 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:42:10 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.502239629s
2021/04/17 11:42:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:11 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:42:11 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:42:11 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:42:11 [INFO] raft: pipelining replication to peer {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 11:42:11 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:42:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:41004->10.242.244.80:8301: i/o timeout
2021/04/17 11:42:12 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:42:13 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:42:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:41054->10.242.244.80:8301: i/o timeout
2021/04/17 11:42:15 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:42:16 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:42:16 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:42:16 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:42:16 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:42:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:17 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:42:18 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:42:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:19 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:42:20 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500364037s
2021/04/17 11:42:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:54654->10.242.244.80:8302: i/o timeout
2021/04/17 11:42:22 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:42:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:23 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.958507424s
2021/04/17 11:42:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:25 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.422791195s
2021/04/17 11:42:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:28 [ERR] raft: Failed to pipeline AppendEntries to {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: write tcp 10.242.253.198:35790->10.242.244.80:8300: use of closed network connection
2021/04/17 11:42:28 [INFO] raft: aborting pipeline replication to peer {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 11:42:28 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:42:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:29 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:34830->10.242.244.80:8300: i/o timeout
2021/04/17 11:42:29 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:42:30 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500305436s
2021/04/17 11:42:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:54850->10.242.244.80:8302: i/o timeout
2021/04/17 11:42:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:42:33 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:42:33 [INFO] raft: pipelining replication to peer {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 11:42:33 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:42:33 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:42:33 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:42:35 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500289054s
2021/04/17 11:42:35 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:42:35 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:42:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:41476->10.242.244.80:8301: i/o timeout
2021/04/17 11:42:36 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:42:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:38 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.99331642s
2021/04/17 11:42:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:41536->10.242.244.80:8301: i/o timeout
2021/04/17 11:42:39 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:42:39 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:42:39 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:42:39 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:42:40 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.451500661s
2021/04/17 11:42:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:54934->10.242.244.80:8302: i/o timeout
2021/04/17 11:42:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:42:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:43 [INFO] raft: aborting pipeline replication to peer {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 11:42:43 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:42:43 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:36266->10.242.244.80:8300: i/o timeout
2021/04/17 11:42:44 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:42:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:45 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500273362s
2021/04/17 11:42:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:48 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.999115748s
2021/04/17 11:42:48 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:42:48 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:58960
2021/04/17 11:42:48 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:42:48 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:42:49 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:42:50 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.489346951s
2021/04/17 11:42:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:52 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:42:52 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:42:52 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:42:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:53 [ERR] raft: Failed to AppendEntries to {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:36528->10.242.244.80:8300: i/o timeout
2021/04/17 11:42:53 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:42:54 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:36538->10.242.244.80:8300: i/o timeout
2021/04/17 11:42:54 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:42:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:42:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:55336->10.242.244.80:8302: i/o timeout
2021/04/17 11:42:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:42:57 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:42:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:03 [ERR] memberlist: failed to receive: read tcp 10.242.253.198:8302->10.242.244.80:56524: i/o timeout from=10.242.244.80:56524
2021/04/17 11:43:03 [ERR] memberlist: Failed to send error: write tcp 10.242.253.198:8302->10.242.244.80:56524: i/o timeout from=10.242.244.80:56524
2021/04/17 11:43:03 [ERR] raft: Failed to AppendEntries to {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:36720->10.242.244.80:8300: i/o timeout
2021/04/17 11:43:03 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:43:04 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:36748->10.242.244.80:8300: i/o timeout
2021/04/17 11:43:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:05 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:43:05 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:43:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:13 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:43:13 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:43:13 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:43:13 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:43:13 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:43:13 [ERR] raft: Failed to AppendEntries to {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:36918->10.242.244.80:8300: i/o timeout
2021/04/17 11:43:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:15 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:36964->10.242.244.80:8300: i/o timeout
2021/04/17 11:43:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:42296->10.242.244.80:8301: i/o timeout
2021/04/17 11:43:17 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:43:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:19 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:43:19 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:43:19 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:43:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:42318->10.242.244.80:8301: i/o timeout
2021/04/17 11:43:20 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:43:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:22 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:43:23 [ERR] raft: Failed to AppendEntries to {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:37120->10.242.244.80:8300: i/o timeout
2021/04/17 11:43:23 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:43:23 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:43:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:26 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:37180->10.242.244.80:8300: i/o timeout
2021/04/17 11:43:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:55918->10.242.244.80:8302: i/o timeout
2021/04/17 11:43:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:43:27 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:43:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:28 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:43:28 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:43:31 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:43:31 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 11:43:31 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:43:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:33 [ERR] raft: Failed to AppendEntries to {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:37312->10.242.244.80:8300: i/o timeout
2021/04/17 11:43:33 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:43:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:37 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:37396->10.242.244.80:8300: i/o timeout
2021/04/17 11:43:38 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:43:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:56114->10.242.244.80:8302: i/o timeout
2021/04/17 11:43:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:43:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:43 [ERR] raft: Failed to AppendEntries to {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:37514->10.242.244.80:8300: i/o timeout
2021/04/17 11:43:43 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:43:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:48 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:37616->10.242.244.80:8300: i/o timeout
2021/04/17 11:43:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:49 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:43:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:56306->10.242.244.80:8302: i/o timeout
2021/04/17 11:43:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:43:52 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:43:52 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:43:52 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:43:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:53 [ERR] raft: Failed to AppendEntries to {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:37706->10.242.244.80:8300: i/o timeout
2021/04/17 11:43:54 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:43:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:43:59 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:37822->10.242.244.80:8300: i/o timeout
2021/04/17 11:44:00 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:44:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:44:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:44:04 [ERR] raft: Failed to AppendEntries to {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:37898->10.242.244.80:8300: i/o timeout
2021/04/17 11:44:04 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:44:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:44:05 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:44:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:44:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:44:10 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:38038->10.242.244.80:8300: i/o timeout
2021/04/17 11:44:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:44:10 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:44:10 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:44:12 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:44:13 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:44:14 [ERR] raft: Failed to AppendEntries to {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:38104->10.242.244.80:8300: i/o timeout
2021/04/17 11:44:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:44:16 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:44:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:44:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:44:19 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:44:19 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:44:19 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:44:20 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 11:44:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:44:21 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:44:22 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500236101s
2021/04/17 11:44:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:44:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:44:25 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.918007448s
2021/04/17 11:44:26 [ERR] raft: Failed to AppendEntries to {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:38326->10.242.244.80:8300: i/o timeout
2021/04/17 11:44:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:44:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:57072->10.242.244.80:8302: i/o timeout
2021/04/17 11:44:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:44:27 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.41351549s
2021/04/17 11:44:28 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:44:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:44:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:44:31 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:38256->10.242.244.80:8300: i/o timeout
2021/04/17 11:44:31 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:44:32 [ERR] raft: peer {Nonvoter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300} has newer term, stopping replication
2021/04/17 11:44:32 [INFO] raft: Node at 10.242.253.198:8300 [Follower] entering Follower state (Leader: "")
2021/04/17 11:44:32 [INFO] raft: aborting pipeline replication to peer {Voter 38d8f917-7a55-e6fb-c4e2-f154d8409d0c 10.242.146.73:8300}
2021/04/17 11:44:32 [INFO] consul: cluster leadership lost
2021/04/17 11:44:32 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:44:32 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:44:39 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/17 11:44:39 [INFO] raft: Node at 10.242.253.198:8300 [Candidate] entering Candidate state in term 18
2021/04/17 11:44:39 [INFO] raft: Election won. Tally: 2
2021/04/17 11:44:39 [INFO] raft: Node at 10.242.253.198:8300 [Leader] entering Leader state
2021/04/17 11:44:39 [INFO] consul: cluster leadership acquired
2021/04/17 11:44:39 [INFO] raft: Added peer 38d8f917-7a55-e6fb-c4e2-f154d8409d0c, starting replication
2021/04/17 11:44:39 [INFO] raft: Added peer a06ee723-0a7a-bed8-b418-2a3a48a30906, starting replication
2021/04/17 11:44:39 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/17 11:44:39 [INFO] raft: pipelining replication to peer {Voter 38d8f917-7a55-e6fb-c4e2-f154d8409d0c 10.242.146.73:8300}
2021/04/17 11:44:39 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:44:39 [WARN] raft: AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300} rejected, sending older logs (next: 896)
2021/04/17 11:44:39 [INFO] raft: pipelining replication to peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 11:44:44 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:44:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:00 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:50:00 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500384385s
2021/04/17 11:50:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:49898->10.242.244.80:8301: i/o timeout
2021/04/17 11:50:02 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:50:03 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.957138839s
2021/04/17 11:50:04 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:50:06 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:50:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:50036->10.242.244.80:8301: i/o timeout
2021/04/17 11:50:09 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:50:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:09 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500548437s
2021/04/17 11:50:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:35300->10.242.244.80:8302: i/o timeout
2021/04/17 11:50:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:50:13 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500284118s
2021/04/17 11:50:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:50112->10.242.244.80:8301: i/o timeout
2021/04/17 11:50:13 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:50:13 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:50:13 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:50:13 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:50:13 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:50:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:15 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.971759867s
2021/04/17 11:50:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:18 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:50:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:20 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500131482s
2021/04/17 11:50:20 [INFO] raft: aborting pipeline replication to peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 11:50:20 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:50:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:21 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:50:21 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:32936
2021/04/17 11:50:22 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:50:22 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:50:23 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500210069s
2021/04/17 11:50:24 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:50:24 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:50:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:25 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.979235056s
2021/04/17 11:50:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:35554->10.242.244.80:8302: i/o timeout
2021/04/17 11:50:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:50:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:28 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.451156398s
2021/04/17 11:50:28 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:38638->10.242.244.80:8300: i/o timeout
2021/04/17 11:50:29 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:50:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:30 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:45148->10.242.244.80:8300: i/o timeout
2021/04/17 11:50:30 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:50:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:35700->10.242.244.80:8302: i/o timeout
2021/04/17 11:50:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:50:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:38 [ERR] memberlist: Push/Pull with dev-consul-server-2.dc1 failed: read tcp 10.242.253.198:35274->10.242.244.80:8302: i/o timeout
2021/04/17 11:50:39 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:45282->10.242.244.80:8300: i/o timeout
2021/04/17 11:50:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:39 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:50:40 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:45300->10.242.244.80:8300: i/o timeout
2021/04/17 11:50:40 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:50:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:35852->10.242.244.80:8302: i/o timeout
2021/04/17 11:50:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:50:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:49 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:45434->10.242.244.80:8300: i/o timeout
2021/04/17 11:50:50 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:50:50 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:45444->10.242.244.80:8300: i/o timeout
2021/04/17 11:50:50 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:50:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:52 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:50:52 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:50:52 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:50:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:36074->10.242.244.80:8302: i/o timeout
2021/04/17 11:50:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:50:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:50:57 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:50:57 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:50:57 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:50:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:50924->10.242.244.80:8301: i/o timeout
2021/04/17 11:51:00 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:51:00 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:45654->10.242.244.80:8300: i/o timeout
2021/04/17 11:51:00 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:45656->10.242.244.80:8300: i/o timeout
2021/04/17 11:51:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:03 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:51:03 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:51:03 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:51:03 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:51:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:10 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:45832->10.242.244.80:8300: i/o timeout
2021/04/17 11:51:10 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:51:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:11 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:45836->10.242.244.80:8300: i/o timeout
2021/04/17 11:51:12 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:51:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:19 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:51:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:20 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:46040->10.242.244.80:8300: i/o timeout
2021/04/17 11:51:21 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:51:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:22 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:46054->10.242.244.80:8300: i/o timeout
2021/04/17 11:51:23 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:51:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:24 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:51:24 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:51:24 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 11:51:24 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:51:24 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:51:24 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:51:24 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:51:24 [INFO] raft: pipelining replication to peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 11:51:28 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:51:28 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 11:51:28 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:51:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:51508->10.242.244.80:8301: i/o timeout
2021/04/17 11:51:29 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:51:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:51540->10.242.244.80:8301: i/o timeout
2021/04/17 11:51:32 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:51:32 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 11:51:32 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.50031155s
2021/04/17 11:51:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:51558->10.242.244.80:8301: i/o timeout
2021/04/17 11:51:33 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:51:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:34 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500299017s
2021/04/17 11:51:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:35 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:51:35 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:51:35 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:51:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:37 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.949438235s
2021/04/17 11:51:37 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:51:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:39 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.406808398s
2021/04/17 11:51:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:41 [ERR] memberlist: failed to receive: read tcp 10.242.253.198:8301->10.242.244.80:42662: i/o timeout from=10.242.244.80:42662
2021/04/17 11:51:41 [ERR] memberlist: Failed to send error: write tcp 10.242.253.198:8301->10.242.244.80:42662: i/o timeout from=10.242.244.80:42662
2021/04/17 11:51:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:36978->10.242.244.80:8302: i/o timeout
2021/04/17 11:51:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:51:42 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500350661s
2021/04/17 11:51:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:43 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:51:44 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:51:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:46 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:51:46 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:51:46 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:51:47 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:51:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:49 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500298847s
2021/04/17 11:51:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:51942->10.242.244.80:8301: i/o timeout
2021/04/17 11:51:51 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:51:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:37174->10.242.244.80:8302: i/o timeout
2021/04/17 11:51:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:51:52 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.966561196s
2021/04/17 11:51:52 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:51:52 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:51:52 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:51:52 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:51:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:53 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:51:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:51988->10.242.244.80:8301: i/o timeout
2021/04/17 11:51:54 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:51:54 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:51:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:56 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.508446254s
2021/04/17 11:51:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:57 [ERR] raft: Failed to pipeline AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: write tcp 10.242.253.198:46258->10.242.244.80:8300: use of closed network connection
2021/04/17 11:51:57 [INFO] raft: aborting pipeline replication to peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 11:51:57 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:51:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:51:59 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500282518s
2021/04/17 11:52:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:52:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:37352->10.242.244.80:8302: i/o timeout
2021/04/17 11:52:02 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:52:02 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.998485835s
2021/04/17 11:52:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:52:04 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:52:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:52:06 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:52:06 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:52:07 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 11:52:07 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:52:07 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:52:07 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:52:07 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:46958->10.242.244.80:8300: i/o timeout
2021/04/17 11:52:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:52:09 [INFO] raft: pipelining replication to peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 11:52:10 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:52:10 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 11:52:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52316->10.242.244.80:8301: i/o timeout
2021/04/17 11:52:11 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:52:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:52:13 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:52:13 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:52:13 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:52:13 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:52:14 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:52:14 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:52:14 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:52:16 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:52:16 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 11:52:16 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:52:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:52:19 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/17 11:52:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:52:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:52:22 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/17 11:52:24 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:52:26 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/17 11:52:28 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:52:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52678->10.242.244.80:8301: i/o timeout
2021/04/17 11:52:29 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:52:29 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500290423s
2021/04/17 11:52:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:52:31 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:52:31 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:52:31 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:52:31 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:52:31 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:52:32 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:52:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52712->10.242.244.80:8301: i/o timeout
2021/04/17 11:52:32 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:52:32 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:52:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:52:33 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.505219461s
2021/04/17 11:52:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:52:36 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.934266792s
2021/04/17 11:52:37 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:52:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:38044->10.242.244.80:8302: i/o timeout
2021/04/17 11:52:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:52:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:52:38 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:52:38 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:52:39 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:52:40 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:52:40 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:52:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:52:41 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:52:42 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:52:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:52:43 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:52:44 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:52:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:52:46 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500276915s
2021/04/17 11:52:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:52:48 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:52:49 [INFO] raft: aborting pipeline replication to peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 11:52:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:52:49 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:52:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:52:51 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500263982s
2021/04/17 11:52:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:38342->10.242.244.80:8302: i/o timeout
2021/04/17 11:52:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:52:54 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.957402047s
2021/04/17 11:52:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:52:56 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.434072845s
2021/04/17 11:52:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:38420->10.242.244.80:8302: i/o timeout
2021/04/17 11:52:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:52:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:52:58 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:47174->10.242.244.80:8300: i/o timeout
2021/04/17 11:52:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:52:59 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:47984->10.242.244.80:8300: i/o timeout
2021/04/17 11:52:59 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:52:59 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:53:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:53:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:53:05 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:53:05 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:53:05 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:53:05 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:53:05 [INFO] raft: pipelining replication to peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 11:53:06 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:53:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:53:10 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:53:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:53:13 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:53:34 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:53:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:53942->10.242.244.80:8301: i/o timeout
2021/04/17 11:53:35 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:53:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:53:36 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500303843s
2021/04/17 11:53:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:54092->10.242.244.80:8301: i/o timeout
2021/04/17 11:53:44 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:53:44 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:53:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:53:45 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500329203s
2021/04/17 11:53:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:53:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:54166->10.242.244.80:8301: i/o timeout
2021/04/17 11:53:48 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:53:48 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:53:48 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:53:48 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:53:48 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:53:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:53:50 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500300971s
2021/04/17 11:53:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:53:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:39490->10.242.244.80:8302: i/o timeout
2021/04/17 11:53:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:53:53 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.950608111s
2021/04/17 11:53:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:53:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:53:55 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.41381173s
2021/04/17 11:53:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:53:57 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:48158->10.242.244.80:8300: i/o timeout
2021/04/17 11:53:57 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:53:58 [INFO] raft: aborting pipeline replication to peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 11:53:58 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:53:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:00 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500252342s
2021/04/17 11:54:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:39658->10.242.244.80:8302: i/o timeout
2021/04/17 11:54:02 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:54:02 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:54:02 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:38596
2021/04/17 11:54:03 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:54:03 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:54:03 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:54:03 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:54:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:03 [INFO] raft: pipelining replication to peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 11:54:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:54524->10.242.244.80:8301: i/o timeout
2021/04/17 11:54:05 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:54:06 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500397011s
2021/04/17 11:54:06 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:54:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:54564->10.242.244.80:8301: i/o timeout
2021/04/17 11:54:08 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:54:08 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:54:08 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:54:08 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:54:08 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:54:08 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.99056704s
2021/04/17 11:54:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:11 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.391916522s
2021/04/17 11:54:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:13 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:49274->10.242.244.80:8300: i/o timeout
2021/04/17 11:54:14 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500245585s
2021/04/17 11:54:14 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:54:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:16 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.964676981s
2021/04/17 11:54:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:17 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:54:18 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:54:18 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:54:18 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:54:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:54822->10.242.244.80:8301: i/o timeout
2021/04/17 11:54:19 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:54:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:19 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.462210001s
2021/04/17 11:54:21 [ERR] raft: Failed to pipeline AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: write tcp 10.242.253.198:49284->10.242.244.80:8300: use of closed network connection
2021/04/17 11:54:21 [INFO] raft: aborting pipeline replication to peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 11:54:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:54856->10.242.244.80:8301: i/o timeout
2021/04/17 11:54:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:54:22 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:54:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:23 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:54:23 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:54:23 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:54:24 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500265907s
2021/04/17 11:54:24 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:49588->10.242.244.80:8300: i/o timeout
2021/04/17 11:54:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:25 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:54:26 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.999564782s
2021/04/17 11:54:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:40156->10.242.244.80:8302: i/o timeout
2021/04/17 11:54:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:54:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:29 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.466728726s
2021/04/17 11:54:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:32 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:49750->10.242.244.80:8300: i/o timeout
2021/04/17 11:54:32 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:54:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:35 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:49816->10.242.244.80:8300: i/o timeout
2021/04/17 11:54:36 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:54:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:42 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:49926->10.242.244.80:8300: i/o timeout
2021/04/17 11:54:42 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:54:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:40454->10.242.244.80:8302: i/o timeout
2021/04/17 11:54:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:54:42 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:54:42 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:54:42 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:54:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:46 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:50042->10.242.244.80:8300: i/o timeout
2021/04/17 11:54:47 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:54:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:52 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:50130->10.242.244.80:8300: i/o timeout
2021/04/17 11:54:52 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:54:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:40526->10.242.244.80:8302: i/o timeout
2021/04/17 11:54:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:54:52 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:54:52 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:54:54 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:54:54 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:54:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:57 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:50224->10.242.244.80:8300: i/o timeout
2021/04/17 11:54:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:54:58 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:54:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:02 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:50346->10.242.244.80:8300: i/o timeout
2021/04/17 11:55:02 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:55:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:40940->10.242.244.80:8302: i/o timeout
2021/04/17 11:55:07 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:55:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:08 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:50464->10.242.244.80:8300: i/o timeout
2021/04/17 11:55:09 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:55:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:12 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:50504->10.242.244.80:8300: i/o timeout
2021/04/17 11:55:12 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:55:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:41040->10.242.244.80:8302: i/o timeout
2021/04/17 11:55:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:55:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:17 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:55:19 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:50682->10.242.244.80:8300: i/o timeout
2021/04/17 11:55:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:20 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:55:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:22 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:50716->10.242.244.80:8300: i/o timeout
2021/04/17 11:55:22 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:55:22 [ERR] memberlist: failed to receive: read tcp 10.242.253.198:8302->10.242.244.80:37806: i/o timeout from=10.242.244.80:37806
2021/04/17 11:55:22 [ERR] memberlist: Failed to send error: write tcp 10.242.253.198:8302->10.242.244.80:37806: i/o timeout from=10.242.244.80:37806
2021/04/17 11:55:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:41318->10.242.244.80:8302: i/o timeout
2021/04/17 11:55:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:55:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:30 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:50906->10.242.244.80:8300: i/o timeout
2021/04/17 11:55:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:31 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:55:32 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:55:32 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:55:32 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:55:32 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:50928->10.242.244.80:8300: i/o timeout
2021/04/17 11:55:32 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:55:32 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:55:32 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:55:32 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:55:32 [ERR] yamux: Failed to read stream data: read tcp 10.242.253.198:8300->10.242.244.80:40924: use of closed network connection
2021/04/17 11:55:32 [WARN] yamux: failed to send go away: session shutdown
2021/04/17 11:55:32 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:40924
2021/04/17 11:55:34 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:55:34 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 11:55:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:41524->10.242.244.80:8302: i/o timeout
2021/04/17 11:55:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:55:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:41 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:51086->10.242.244.80:8300: i/o timeout
2021/04/17 11:55:42 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:51104->10.242.244.80:8300: i/o timeout
2021/04/17 11:55:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:43 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:55:43 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:55:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:50 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:55:50 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:55:50 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:55:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:53 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:51320->10.242.244.80:8300: i/o timeout
2021/04/17 11:55:53 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:51324->10.242.244.80:8300: i/o timeout
2021/04/17 11:55:54 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:55:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:56670->10.242.244.80:8301: i/o timeout
2021/04/17 11:55:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:55:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:56 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:55:56 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:55:56 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:55:56 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:55:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:55:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:04 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:51556->10.242.244.80:8300: i/o timeout
2021/04/17 11:56:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:06 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:51598->10.242.244.80:8300: i/o timeout
2021/04/17 11:56:07 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:56:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:12 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:56:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:13 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:56:13 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:56:16 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:56:17 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:51796->10.242.244.80:8300: i/o timeout
2021/04/17 11:56:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:19 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:56:19 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:41450
2021/04/17 11:56:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:22 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:56:22 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:51886->10.242.244.80:8300: i/o timeout
2021/04/17 11:56:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:32 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:52080->10.242.244.80:8300: i/o timeout
2021/04/17 11:56:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:33 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:56:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:42 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:56:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:43 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:52314->10.242.244.80:8300: i/o timeout
2021/04/17 11:56:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:52 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:52466->10.242.244.80:8300: i/o timeout
2021/04/17 11:56:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:54 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:56:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:55 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:56:55 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:56:57 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:56:57 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:56:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:56:58 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 11:56:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:56:59 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:56:59 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:56:59 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:57:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:57934->10.242.244.80:8301: i/o timeout
2021/04/17 11:57:02 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:57:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:04 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:52710->10.242.244.80:8300: i/o timeout
2021/04/17 11:57:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:58020->10.242.244.80:8301: i/o timeout
2021/04/17 11:57:05 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:57:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:06 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:57:06 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:57:06 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:57:06 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:57:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:12 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:52862->10.242.244.80:8300: i/o timeout
2021/04/17 11:57:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:16 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:57:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:43438->10.242.244.80:8302: i/o timeout
2021/04/17 11:57:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:57:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:23 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:57:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:26 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:53112->10.242.244.80:8300: i/o timeout
2021/04/17 11:57:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:43640->10.242.244.80:8302: i/o timeout
2021/04/17 11:57:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:57:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:33 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:53262->10.242.244.80:8300: i/o timeout
2021/04/17 11:57:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:35 [ERR] memberlist: Push/Pull with dev-consul-server-2.dc1 failed: read tcp 10.242.253.198:43270->10.242.244.80:8302: i/o timeout
2021/04/17 11:57:36 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:57:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:43844->10.242.244.80:8302: i/o timeout
2021/04/17 11:57:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:57:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:37 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:57:37 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:57:40 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:57:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:42 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:57:42 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:57:42 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:57:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:43 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:57:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:46 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:53530->10.242.244.80:8300: i/o timeout
2021/04/17 11:57:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:53 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:53630->10.242.244.80:8300: i/o timeout
2021/04/17 11:57:54 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:57:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:57:57 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:57:57 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:57:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:03 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:58:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:07 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:53930->10.242.244.80:8300: i/o timeout
2021/04/17 11:58:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:13 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:54052->10.242.244.80:8300: i/o timeout
2021/04/17 11:58:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:19 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:58:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:19 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:58:19 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:58:22 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:58:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:23 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:58:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:29 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:54364->10.242.244.80:8300: i/o timeout
2021/04/17 11:58:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:33 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:54424->10.242.244.80:8300: i/o timeout
2021/04/17 11:58:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:40 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:58:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:44 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:58:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:50 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:54750->10.242.244.80:8300: i/o timeout
2021/04/17 11:58:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:54 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:54796->10.242.244.80:8300: i/o timeout
2021/04/17 11:58:54 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:58:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:58:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:00 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:59:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:01 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:59:01 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:59:04 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:59:04 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:59:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:10 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:55128->10.242.244.80:8300: i/o timeout
2021/04/17 11:59:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:14 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:55214->10.242.244.80:8300: i/o timeout
2021/04/17 11:59:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:21 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:59:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:24 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:59:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:31 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:55546->10.242.244.80:8300: i/o timeout
2021/04/17 11:59:31 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:59:31 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:59:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:34 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:55586->10.242.244.80:8300: i/o timeout
2021/04/17 11:59:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:38 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:59:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:42 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:59:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:43 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:59:43 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:59:44 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 11:59:46 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:59:46 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 11:59:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:46312->10.242.244.80:8302: i/o timeout
2021/04/17 11:59:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:59:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:52 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:55934->10.242.244.80:8300: i/o timeout
2021/04/17 11:59:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:54 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:55954->10.242.244.80:8300: i/o timeout
2021/04/17 11:59:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:46530->10.242.244.80:8302: i/o timeout
2021/04/17 11:59:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:59:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:59:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:04 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:00:05 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:00:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:46614->10.242.244.80:8302: i/o timeout
2021/04/17 12:00:07 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:00:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:12 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 12:00:12 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:00:12 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:00:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:14 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:56350->10.242.244.80:8300: i/o timeout
2021/04/17 12:00:15 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:56374->10.242.244.80:8300: i/o timeout
2021/04/17 12:00:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:46894->10.242.244.80:8302: i/o timeout
2021/04/17 12:00:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:00:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:24 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:00:25 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:00:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:25 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:00:25 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:00:25 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:00:28 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:00:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:35 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:56760->10.242.244.80:8300: i/o timeout
2021/04/17 12:00:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:35 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:56764->10.242.244.80:8300: i/o timeout
2021/04/17 12:00:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:45 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:00:46 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:00:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:48 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:00:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:54 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:00:54 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:00:54 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 12:00:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:55 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:57128->10.242.244.80:8300: i/o timeout
2021/04/17 12:00:56 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:57154->10.242.244.80:8300: i/o timeout
2021/04/17 12:00:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:00:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:34274->10.242.244.80:8301: i/o timeout
2021/04/17 12:00:58 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:00:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:00 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:01:00 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:01:00 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 12:01:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:34320->10.242.244.80:8301: i/o timeout
2021/04/17 12:01:01 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:01:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:05 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:01:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:07 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:01:07 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:01:07 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:01:10 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:01:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:15 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:57546->10.242.244.80:8300: i/o timeout
2021/04/17 12:01:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:17 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:57580->10.242.244.80:8300: i/o timeout
2021/04/17 12:01:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:26 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:01:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:28 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:01:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:36 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:57924->10.242.244.80:8300: i/o timeout
2021/04/17 12:01:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:38 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:57984->10.242.244.80:8300: i/o timeout
2021/04/17 12:01:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:46 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:01:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:49 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:01:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:49 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:01:49 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:01:52 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:01:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:56 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:58302->10.242.244.80:8300: i/o timeout
2021/04/17 12:01:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:01:59 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:58398->10.242.244.80:8300: i/o timeout
2021/04/17 12:01:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:06 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:02:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:10 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:02:11 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:02:11 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:02:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:11 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:02:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:13 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500169354s
2021/04/17 12:02:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:16 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.996085827s
2021/04/17 12:02:16 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:58712->10.242.244.80:8300: i/o timeout
2021/04/17 12:02:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:18 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.466857463s
2021/04/17 12:02:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:21 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:58774->10.242.244.80:8300: i/o timeout
2021/04/17 12:02:22 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:02:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:26 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:02:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:49402->10.242.244.80:8302: i/o timeout
2021/04/17 12:02:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:02:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:31 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:02:31 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:02:32 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:59030->10.242.244.80:8300: i/o timeout
2021/04/17 12:02:33 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:02:34 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:02:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:36 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:59096->10.242.244.80:8300: i/o timeout
2021/04/17 12:02:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:49508->10.242.244.80:8302: i/o timeout
2021/04/17 12:02:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:02:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:41 [ERR] memberlist: failed to receive: read tcp 10.242.253.198:8302->10.242.244.80:42304: i/o timeout from=10.242.244.80:42304
2021/04/17 12:02:41 [ERR] memberlist: Failed to send error: write tcp 10.242.253.198:8302->10.242.244.80:42304: i/o timeout from=10.242.244.80:42304
2021/04/17 12:02:41 [ERR] memberlist: failed to receive: read tcp 10.242.253.198:8302->10.242.244.80:42306: i/o timeout from=10.242.244.80:42306
2021/04/17 12:02:41 [ERR] memberlist: Failed to send error: write tcp 10.242.253.198:8302->10.242.244.80:42306: i/o timeout from=10.242.244.80:42306
2021/04/17 12:02:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:43 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:59248->10.242.244.80:8300: i/o timeout
2021/04/17 12:02:44 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:02:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:47 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:02:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:49772->10.242.244.80:8302: i/o timeout
2021/04/17 12:02:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:02:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:47 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 12:02:47 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:02:47 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:02:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:54 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:02:54 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:02:54 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:59416->10.242.244.80:8300: i/o timeout
2021/04/17 12:02:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:55 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:02:57 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:59472->10.242.244.80:8300: i/o timeout
2021/04/17 12:02:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:02:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:05 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:59658->10.242.244.80:8300: i/o timeout
2021/04/17 12:03:06 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:03:07 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:03:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:13 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:03:13 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:03:14 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 12:03:14 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 12:03:14 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.146.73
2021/04/17 12:03:14 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.146.73:8300) (DC: dc1)
2021/04/17 12:03:14 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/17 12:03:14 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.146.73
2021/04/17 12:03:14 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.146.73:8300) (DC: dc1)
2021/04/17 12:03:14 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/17 12:03:14 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:03:15 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:03:15 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:03:15 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:03:16 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:03:16 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 12:03:17 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:59908->10.242.244.80:8300: i/o timeout
2021/04/17 12:03:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:17 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500349335s
2021/04/17 12:03:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:20 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.977894941s
2021/04/17 12:03:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:23 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:03:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:24 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500266172s
2021/04/17 12:03:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:26 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:03:26 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:03:26 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 12:03:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:50518->10.242.244.80:8302: i/o timeout
2021/04/17 12:03:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:03:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:29 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500248148s
2021/04/17 12:03:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:37210->10.242.244.80:8301: i/o timeout
2021/04/17 12:03:31 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:03:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:32 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.998966647s
2021/04/17 12:03:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:37242->10.242.244.80:8301: i/o timeout
2021/04/17 12:03:34 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:03:34 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:03:34 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:03:34 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 12:03:34 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.447860582s
2021/04/17 12:03:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:37 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:59886->10.242.244.80:8300: i/o timeout
2021/04/17 12:03:38 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:60298->10.242.244.80:8300: i/o timeout
2021/04/17 12:03:38 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:03:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:50890->10.242.244.80:8302: i/o timeout
2021/04/17 12:03:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:03:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:47 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:03:48 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:60532->10.242.244.80:8300: i/o timeout
2021/04/17 12:03:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:03:49 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:03:51 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:03:51 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:03:51 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:03:51 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 12:03:51 [INFO] raft: pipelining replication to peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 12:04:34 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:04:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:04:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:38778->10.242.244.80:8301: i/o timeout
2021/04/17 12:04:51 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:04:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:04:51 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500396879s
2021/04/17 12:04:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:04:54 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.990571704s
2021/04/17 12:04:54 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:04:54 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:04:54 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 12:04:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:38836->10.242.244.80:8301: i/o timeout
2021/04/17 12:04:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:04:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:04:56 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.467650979s
2021/04/17 12:04:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52312->10.242.244.80:8302: i/o timeout
2021/04/17 12:04:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:04:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:04:59 [INFO] raft: aborting pipeline replication to peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 12:04:59 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:04:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:04:59 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:60734->10.242.244.80:8300: i/o timeout
2021/04/17 12:04:59 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:05:01 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:05:01 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:05:01 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500264658s
2021/04/17 12:05:02 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:05:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:04 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 5.000161951s
2021/04/17 12:05:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:06 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.481009022s
2021/04/17 12:05:07 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/17 12:05:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:09 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:33822->10.242.244.80:8300: i/o timeout
2021/04/17 12:05:09 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:05:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:09 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:33836->10.242.244.80:8300: i/o timeout
2021/04/17 12:05:10 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:05:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:19 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:34020->10.242.244.80:8300: i/o timeout
2021/04/17 12:05:19 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:05:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:20 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:34044->10.242.244.80:8300: i/o timeout
2021/04/17 12:05:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:21 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:05:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52822->10.242.244.80:8302: i/o timeout
2021/04/17 12:05:22 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:05:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:27 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 12:05:27 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:05:27 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:05:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:29 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:34226->10.242.244.80:8300: i/o timeout
2021/04/17 12:05:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:29 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:05:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:31 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:34260->10.242.244.80:8300: i/o timeout
2021/04/17 12:05:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52890->10.242.244.80:8302: i/o timeout
2021/04/17 12:05:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:05:32 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:05:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:34 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:05:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:39 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:34406->10.242.244.80:8300: i/o timeout
2021/04/17 12:05:39 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:05:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:41 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:05:41 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:05:41 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:05:41 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:46246
2021/04/17 12:05:42 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:34442->10.242.244.80:8300: i/o timeout
2021/04/17 12:05:43 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:05:44 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:05:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:49 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:34604->10.242.244.80:8300: i/o timeout
2021/04/17 12:05:49 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:05:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:53 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:34652->10.242.244.80:8300: i/o timeout
2021/04/17 12:05:54 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:05:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:05:59 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:34816->10.242.244.80:8300: i/o timeout
2021/04/17 12:05:59 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:06:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:04 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:34880->10.242.244.80:8300: i/o timeout
2021/04/17 12:06:05 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:06:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:09 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:34984->10.242.244.80:8300: i/o timeout
2021/04/17 12:06:09 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:06:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:15 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:35092->10.242.244.80:8300: i/o timeout
2021/04/17 12:06:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:16 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:06:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:19 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:35194->10.242.244.80:8300: i/o timeout
2021/04/17 12:06:20 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:06:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:23 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:06:23 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:06:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:06:26 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:35304->10.242.244.80:8300: i/o timeout
2021/04/17 12:06:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:28 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:06:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:30 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:35412->10.242.244.80:8300: i/o timeout
2021/04/17 12:06:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:31 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:06:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:38 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:35550->10.242.244.80:8300: i/o timeout
2021/04/17 12:06:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:41 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:35594->10.242.244.80:8300: i/o timeout
2021/04/17 12:06:42 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:06:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:44 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:06:45 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:06:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:52 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:35804->10.242.244.80:8300: i/o timeout
2021/04/17 12:06:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:54 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:35830->10.242.244.80:8300: i/o timeout
2021/04/17 12:06:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:57 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:06:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:06:59 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:07:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:05 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:07:05 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:07:07 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:36126->10.242.244.80:8300: i/o timeout
2021/04/17 12:07:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:07:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:09 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:36154->10.242.244.80:8300: i/o timeout
2021/04/17 12:07:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:18 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:07:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:19 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:07:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:28 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:36540->10.242.244.80:8300: i/o timeout
2021/04/17 12:07:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:29 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:36562->10.242.244.80:8300: i/o timeout
2021/04/17 12:07:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:39 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:07:39 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:07:39 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 12:07:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:42084->10.242.244.80:8301: i/o timeout
2021/04/17 12:07:43 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:07:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:42118->10.242.244.80:8301: i/o timeout
2021/04/17 12:07:46 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:07:46 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:07:46 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:07:46 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 12:07:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:47 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:07:47 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:07:49 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:36928->10.242.244.80:8300: i/o timeout
2021/04/17 12:07:50 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:36940->10.242.244.80:8300: i/o timeout
2021/04/17 12:07:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:07:50 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 12:07:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:07:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:00 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:08:00 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:08:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:10 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:37322->10.242.244.80:8300: i/o timeout
2021/04/17 12:08:10 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:37324->10.242.244.80:8300: i/o timeout
2021/04/17 12:08:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:11 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 12:08:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:14 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:08:15 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:08:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:19 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:08:19 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:49694
2021/04/17 12:08:20 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:08:21 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:08:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:29 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:08:29 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:08:30 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:37732->10.242.244.80:8300: i/o timeout
2021/04/17 12:08:31 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:37742->10.242.244.80:8300: i/o timeout
2021/04/17 12:08:32 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:08:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:40 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:08:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:41 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:08:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:50 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:38108->10.242.244.80:8300: i/o timeout
2021/04/17 12:08:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:51 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:38134->10.242.244.80:8300: i/o timeout
2021/04/17 12:08:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:08:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:00 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:09:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:03 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:09:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:09 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 12:09:10 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:38484->10.242.244.80:8300: i/o timeout
2021/04/17 12:09:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:11 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:09:11 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:09:13 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:38510->10.242.244.80:8300: i/o timeout
2021/04/17 12:09:14 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:09:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:21 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:09:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:24 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:09:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:31 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:38888->10.242.244.80:8300: i/o timeout
2021/04/17 12:09:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:34 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:38934->10.242.244.80:8300: i/o timeout
2021/04/17 12:09:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:41 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:09:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:45 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:09:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:51 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:39268->10.242.244.80:8300: i/o timeout
2021/04/17 12:09:52 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:09:52 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:50692
2021/04/17 12:09:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:53 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:09:53 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:09:54 [ERR] memberlist: failed to receive: read tcp 10.242.253.198:8301->10.242.244.80:54948: i/o timeout from=10.242.244.80:54948
2021/04/17 12:09:54 [ERR] memberlist: Failed to send error: write tcp 10.242.253.198:8301->10.242.244.80:54948: i/o timeout from=10.242.244.80:54948
2021/04/17 12:09:55 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:39322->10.242.244.80:8300: i/o timeout
2021/04/17 12:09:56 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:09:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:09:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:01 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:10:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:06 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:10:06 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 12:10:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:11 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:39648->10.242.244.80:8300: i/o timeout
2021/04/17 12:10:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:16 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:39746->10.242.244.80:8300: i/o timeout
2021/04/17 12:10:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:21 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:10:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:26 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:10:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:31 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:40056->10.242.244.80:8300: i/o timeout
2021/04/17 12:10:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:35 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:10:35 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:10:36 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:40128->10.242.244.80:8300: i/o timeout
2021/04/17 12:10:38 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:10:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:39 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-zlv6m 10.242.253.199
2021/04/17 12:10:39 [INFO] consul: member 'dev-consul-7467bf7b46-zlv6m' failed, marking health critical
2021/04/17 12:10:39 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:10:39 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:10:39 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-zlv6m 10.242.253.199
2021/04/17 12:10:39 [INFO] consul: member 'dev-consul-7467bf7b46-zlv6m' joined, marking health alive
2021/04/17 12:10:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:42 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:10:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:45 [ERR] raft: peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300} has newer term, stopping replication
2021/04/17 12:10:45 [INFO] raft: Node at 10.242.253.198:8300 [Follower] entering Follower state (Leader: "")
2021/04/17 12:10:45 [INFO] raft: aborting pipeline replication to peer {Voter 38d8f917-7a55-e6fb-c4e2-f154d8409d0c 10.242.146.73:8300}
2021/04/17 12:10:45 [INFO] consul: cluster leadership lost
2021/04/17 12:10:45 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:10:45 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:10:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:45602->10.242.244.80:8301: i/o timeout
2021/04/17 12:10:47 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:10:50 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since our last index is greater (1233, 1161)
2021/04/17 12:10:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:45756->10.242.244.80:8301: i/o timeout
2021/04/17 12:10:53 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:10:53 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/17 12:10:53 [INFO] raft: Node at 10.242.253.198:8300 [Candidate] entering Candidate state in term 21
2021/04/17 12:10:53 [INFO] raft: Election won. Tally: 2
2021/04/17 12:10:53 [INFO] raft: Node at 10.242.253.198:8300 [Leader] entering Leader state
2021/04/17 12:10:53 [INFO] raft: Added peer 38d8f917-7a55-e6fb-c4e2-f154d8409d0c, starting replication
2021/04/17 12:10:53 [INFO] raft: Added peer a06ee723-0a7a-bed8-b418-2a3a48a30906, starting replication
2021/04/17 12:10:53 [INFO] consul: cluster leadership acquired
2021/04/17 12:10:53 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/17 12:10:53 [INFO] raft: pipelining replication to peer {Voter 38d8f917-7a55-e6fb-c4e2-f154d8409d0c 10.242.146.73:8300}
2021/04/17 12:10:53 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 12:10:56 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.502535822s
2021/04/17 12:10:56 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 12:10:56 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:10:56 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:10:56 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 12:10:56 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:10:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:59268->10.242.244.80:8302: i/o timeout
2021/04/17 12:10:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:10:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:45832->10.242.244.80:8301: i/o timeout
2021/04/17 12:10:57 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:10:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:10:58 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.975569017s
2021/04/17 12:10:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:01 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.416362893s
2021/04/17 12:11:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:03 [ERR] raft: Failed to make RequestVote RPC to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:40416->10.242.244.80:8300: i/o timeout
2021/04/17 12:11:03 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:40670->10.242.244.80:8300: i/o timeout
2021/04/17 12:11:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:03 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:11:04 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:40678->10.242.244.80:8300: i/o timeout
2021/04/17 12:11:05 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:11:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:59472->10.242.244.80:8302: i/o timeout
2021/04/17 12:11:07 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:11:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:13 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:40854->10.242.244.80:8300: i/o timeout
2021/04/17 12:11:13 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:11:15 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:40900->10.242.244.80:8300: i/o timeout
2021/04/17 12:11:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:16 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:11:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:59632->10.242.244.80:8302: i/o timeout
2021/04/17 12:11:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:11:17 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:11:17 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:11:18 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:11:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:23 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:41038->10.242.244.80:8300: i/o timeout
2021/04/17 12:11:23 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:11:24 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:11:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:26 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:41080->10.242.244.80:8300: i/o timeout
2021/04/17 12:11:26 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:11:27 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 12:11:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:59782->10.242.244.80:8302: i/o timeout
2021/04/17 12:11:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:11:27 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:11:27 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:11:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:33 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:41254->10.242.244.80:8300: i/o timeout
2021/04/17 12:11:33 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:11:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:36 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:41310->10.242.244.80:8300: i/o timeout
2021/04/17 12:11:37 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:11:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:43 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:41444->10.242.244.80:8300: i/o timeout
2021/04/17 12:11:43 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:11:45 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:11:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:47 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:41548->10.242.244.80:8300: i/o timeout
2021/04/17 12:11:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:48 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:11:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:53 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:41628->10.242.244.80:8300: i/o timeout
2021/04/17 12:11:54 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:11:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:11:57 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:11:57 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:11:58 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:41754->10.242.244.80:8300: i/o timeout
2021/04/17 12:11:59 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:12:00 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:12:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:04 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:41844->10.242.244.80:8300: i/o timeout
2021/04/17 12:12:04 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:12:04 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:12:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:09 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:41944->10.242.244.80:8300: i/o timeout
2021/04/17 12:12:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:10 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:12:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:14 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:42038->10.242.244.80:8300: i/o timeout
2021/04/17 12:12:15 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:12:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:20 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:42170->10.242.244.80:8300: i/o timeout
2021/04/17 12:12:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:22 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:12:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:25 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:42226->10.242.244.80:8300: i/o timeout
2021/04/17 12:12:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:26 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:12:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:32 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:42386->10.242.244.80:8300: i/o timeout
2021/04/17 12:12:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:34 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:12:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:36 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:42462->10.242.244.80:8300: i/o timeout
2021/04/17 12:12:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:38 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:12:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:39 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:12:39 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:12:42 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:12:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:44 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:42614->10.242.244.80:8300: i/o timeout
2021/04/17 12:12:45 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:12:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:47 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:12:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:48 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:42722->10.242.244.80:8300: i/o timeout
2021/04/17 12:12:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:52 [ERR] memberlist: failed to receive: read tcp 10.242.253.198:8301->10.242.244.80:56612: i/o timeout from=10.242.244.80:56612
2021/04/17 12:12:52 [ERR] memberlist: Failed to send error: write tcp 10.242.253.198:8301->10.242.244.80:56612: i/o timeout from=10.242.244.80:56612
2021/04/17 12:12:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:54 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:12:55 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 12:12:55 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:12:55 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:12:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:12:55 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 12:12:55 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:12:55 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:12:55 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 12:12:55 [WARN] raft: AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300} rejected, sending older logs (next: 1162)
2021/04/17 12:12:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:12:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:01 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500212316s
2021/04/17 12:13:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:48248->10.242.244.80:8301: i/o timeout
2021/04/17 12:13:02 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:13:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:04 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.994539662s
2021/04/17 12:13:05 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:13:05 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:13:05 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 12:13:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:05 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:42998->10.242.244.80:8300: i/o timeout
2021/04/17 12:13:05 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:13:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:48346->10.242.244.80:8301: i/o timeout
2021/04/17 12:13:06 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:13:06 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.447195949s
2021/04/17 12:13:07 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:13:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:13:09 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:13:09 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:13:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:11 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500258218s
2021/04/17 12:13:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:14 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.959288005s
2021/04/17 12:13:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:16 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.406419367s
2021/04/17 12:13:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:33750->10.242.244.80:8302: i/o timeout
2021/04/17 12:13:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:13:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:19 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:13:19 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:43250->10.242.244.80:8300: i/o timeout
2021/04/17 12:13:19 [INFO] raft: pipelining replication to peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 12:13:19 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:42854->10.242.244.80:8300: i/o timeout
2021/04/17 12:13:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:20 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:13:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:21 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:13:21 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:13:24 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:13:24 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 12:13:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:33964->10.242.244.80:8302: i/o timeout
2021/04/17 12:13:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:13:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:28 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:13:28 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:52716
2021/04/17 12:13:29 [INFO] raft: aborting pipeline replication to peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 12:13:29 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:13:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:30 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:43552->10.242.244.80:8300: i/o timeout
2021/04/17 12:13:30 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:13:31 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500211694s
2021/04/17 12:13:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:34 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.999231055s
2021/04/17 12:13:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:36 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.461326513s
2021/04/17 12:13:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:34074->10.242.244.80:8302: i/o timeout
2021/04/17 12:13:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:13:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:39 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:43716->10.242.244.80:8300: i/o timeout
2021/04/17 12:13:39 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:13:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:40 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:43736->10.242.244.80:8300: i/o timeout
2021/04/17 12:13:41 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:13:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:34342->10.242.244.80:8302: i/o timeout
2021/04/17 12:13:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:13:47 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 12:13:47 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:13:47 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:13:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:49 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:43920->10.242.244.80:8300: i/o timeout
2021/04/17 12:13:49 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:13:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:51 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:43942->10.242.244.80:8300: i/o timeout
2021/04/17 12:13:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:52 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:13:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:13:59 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:44108->10.242.244.80:8300: i/o timeout
2021/04/17 12:13:59 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:13:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:14:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:14:02 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:44172->10.242.244.80:8300: i/o timeout
2021/04/17 12:14:03 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:14:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:14:03 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:14:03 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:14:06 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:14:06 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 12:14:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:14:09 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:44288->10.242.244.80:8300: i/o timeout
2021/04/17 12:14:09 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:14:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:14:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:14:13 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:44384->10.242.244.80:8300: i/o timeout
2021/04/17 12:14:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:14:14 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:14:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:14:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:14:19 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:44496->10.242.244.80:8300: i/o timeout
2021/04/17 12:14:19 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:14:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:14:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:14:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:14:24 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:44562->10.242.244.80:8300: i/o timeout
2021/04/17 12:14:25 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:14:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:14:25 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:14:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:14:29 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:44694->10.242.244.80:8300: i/o timeout
2021/04/17 12:14:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:14:29 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:14:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:14:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:14:35 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:44788->10.242.244.80:8300: i/o timeout
2021/04/17 12:14:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:14:36 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:14:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:14:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:14:39 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:44880->10.242.244.80:8300: i/o timeout
2021/04/17 12:14:40 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:14:41 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:14:41 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:14:41 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 12:14:41 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:14:41 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:14:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:14:42 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 12:14:42 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 12:14:43 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 12:14:43 [ERR] raft: peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300} has newer term, stopping replication
2021/04/17 12:14:43 [INFO] raft: Node at 10.242.253.198:8300 [Follower] entering Follower state (Leader: "")
2021/04/17 12:14:43 [INFO] raft: aborting pipeline replication to peer {Voter 38d8f917-7a55-e6fb-c4e2-f154d8409d0c 10.242.146.73:8300}
2021/04/17 12:14:43 [INFO] consul: cluster leadership lost
2021/04/17 12:14:43 [WARN] consul.coordinate: Batch update failed: node is not the leader
2021/04/17 12:14:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:50268->10.242.244.80:8301: i/o timeout
2021/04/17 12:14:50 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.146.75:54564
2021/04/17 12:14:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:50408->10.242.244.80:8301: i/o timeout
2021/04/17 12:14:51 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:14:51 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:14:51 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:14:52 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/17 12:14:52 [INFO] raft: Node at 10.242.253.198:8300 [Candidate] entering Candidate state in term 22
2021/04/17 12:14:52 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:14:52 [INFO] raft: Election won. Tally: 2
2021/04/17 12:14:52 [INFO] raft: Node at 10.242.253.198:8300 [Leader] entering Leader state
2021/04/17 12:14:52 [INFO] raft: Added peer 38d8f917-7a55-e6fb-c4e2-f154d8409d0c, starting replication
2021/04/17 12:14:52 [INFO] raft: Added peer a06ee723-0a7a-bed8-b418-2a3a48a30906, starting replication
2021/04/17 12:14:52 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:14:52 [INFO] raft: pipelining replication to peer {Voter 38d8f917-7a55-e6fb-c4e2-f154d8409d0c 10.242.146.73:8300}
2021/04/17 12:14:52 [INFO] consul: cluster leadership acquired
2021/04/17 12:14:52 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/17 12:14:52 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 12:14:53 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:14:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:50426->10.242.244.80:8301: i/o timeout
2021/04/17 12:14:54 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:14:54 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.511655286s
2021/04/17 12:14:55 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:14:55 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 12:14:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:14:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:35714->10.242.244.80:8302: i/o timeout
2021/04/17 12:14:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:14:57 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.945281269s
2021/04/17 12:14:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:14:59 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.35633791s
2021/04/17 12:15:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:02 [ERR] raft: Failed to make RequestVote RPC to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:45090->10.242.244.80:8300: i/o timeout
2021/04/17 12:15:02 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:45026->10.242.244.80:8300: i/o timeout
2021/04/17 12:15:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:02 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:15:03 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:45342->10.242.244.80:8300: i/o timeout
2021/04/17 12:15:03 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:15:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:36020->10.242.244.80:8302: i/o timeout
2021/04/17 12:15:12 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:15:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:12 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:45490->10.242.244.80:8300: i/o timeout
2021/04/17 12:15:12 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:15:13 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:45538->10.242.244.80:8300: i/o timeout
2021/04/17 12:15:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:14 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:15:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:36080->10.242.244.80:8302: i/o timeout
2021/04/17 12:15:22 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:15:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:22 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 12:15:22 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:15:22 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:15:22 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:45698->10.242.244.80:8300: i/o timeout
2021/04/17 12:15:22 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:15:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:24 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:45722->10.242.244.80:8300: i/o timeout
2021/04/17 12:15:25 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:15:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:32 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:45908->10.242.244.80:8300: i/o timeout
2021/04/17 12:15:32 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:15:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:35 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:45944->10.242.244.80:8300: i/o timeout
2021/04/17 12:15:36 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:15:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:42 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:46068->10.242.244.80:8300: i/o timeout
2021/04/17 12:15:42 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:15:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:46 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:46168->10.242.244.80:8300: i/o timeout
2021/04/17 12:15:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:47 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:15:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:51 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:15:51 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:15:52 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:46278->10.242.244.80:8300: i/o timeout
2021/04/17 12:15:52 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:15:53 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:15:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:57 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:46346->10.242.244.80:8300: i/o timeout
2021/04/17 12:15:58 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:15:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:15:58 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:15:58 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:15:58 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:15:58 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:15:58 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 12:15:58 [ERR] raft: peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300} has newer term, stopping replication
2021/04/17 12:15:58 [INFO] raft: Node at 10.242.253.198:8300 [Follower] entering Follower state (Leader: "")
2021/04/17 12:15:58 [INFO] raft: aborting pipeline replication to peer {Voter 38d8f917-7a55-e6fb-c4e2-f154d8409d0c 10.242.146.73:8300}
2021/04/17 12:15:58 [INFO] consul: cluster leadership lost
2021/04/17 12:15:58 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 12:15:58 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since our last term is greater (22, 18)
2021/04/17 12:16:07 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/17 12:16:07 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/17 12:16:07 [INFO] raft: Node at 10.242.253.198:8300 [Candidate] entering Candidate state in term 24
2021/04/17 12:16:07 [INFO] raft: Election won. Tally: 2
2021/04/17 12:16:07 [INFO] raft: Node at 10.242.253.198:8300 [Leader] entering Leader state
2021/04/17 12:16:07 [INFO] raft: Added peer 38d8f917-7a55-e6fb-c4e2-f154d8409d0c, starting replication
2021/04/17 12:16:07 [INFO] raft: Added peer a06ee723-0a7a-bed8-b418-2a3a48a30906, starting replication
2021/04/17 12:16:07 [INFO] consul: cluster leadership acquired
2021/04/17 12:16:07 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/17 12:16:07 [INFO] raft: pipelining replication to peer {Voter 38d8f917-7a55-e6fb-c4e2-f154d8409d0c 10.242.146.73:8300}
2021/04/17 12:16:07 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 12:16:07 [WARN] raft: AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300} rejected, sending older logs (next: 1226)
2021/04/17 12:16:07 [INFO] raft: pipelining replication to peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 12:16:16 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:16:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:16:19 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500579202s
2021/04/17 12:16:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52136->10.242.244.80:8301: i/o timeout
2021/04/17 12:16:20 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:16:20 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:16:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:16:22 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 12:16:22 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:16:22 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:16:22 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 12:16:23 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:16:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:16:24 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:16:25 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:16:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:16:26 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500326444s
2021/04/17 12:16:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:16:29 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.968013627s
2021/04/17 12:16:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:16:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:16:31 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.441056495s
2021/04/17 12:16:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:37554->10.242.244.80:8302: i/o timeout
2021/04/17 12:16:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:16:32 [INFO] raft: aborting pipeline replication to peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 12:16:32 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:16:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:16:35 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500280564s
2021/04/17 12:16:35 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:46504->10.242.244.80:8300: i/o timeout
2021/04/17 12:16:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:16:36 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:16:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:16:37 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.931209146s
2021/04/17 12:16:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:16:40 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.391266518s
2021/04/17 12:16:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:16:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:37656->10.242.244.80:8302: i/o timeout
2021/04/17 12:16:42 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:47240->10.242.244.80:8300: i/o timeout
2021/04/17 12:16:42 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:16:43 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:16:43 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:16:43 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 12:16:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:16:44 [INFO] raft: pipelining replication to peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 12:16:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52730->10.242.244.80:8301: i/o timeout
2021/04/17 12:16:51 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:16:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52778->10.242.244.80:8301: i/o timeout
2021/04/17 12:16:54 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:16:56 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:16:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52840->10.242.244.80:8301: i/o timeout
2021/04/17 12:16:57 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:16:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:16:59 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 12:16:59 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:16:59 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:16:59 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 12:16:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:00 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500293589s
2021/04/17 12:17:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:52896->10.242.244.80:8301: i/o timeout
2021/04/17 12:17:00 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:17:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:01 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:17:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:38146->10.242.244.80:8302: i/o timeout
2021/04/17 12:17:02 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:17:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:03 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500408955s
2021/04/17 12:17:04 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:17:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:05 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 12:17:06 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:17:06 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:17:06 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:17:06 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 12:17:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:17:09 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500274643s
2021/04/17 12:17:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:13 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500347959s
2021/04/17 12:17:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:53144->10.242.244.80:8301: i/o timeout
2021/04/17 12:17:13 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:17:13 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 12:17:13 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:17:13 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:17:13 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 12:17:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:14 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:17:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:15 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.966103873s
2021/04/17 12:17:16 [INFO] raft: aborting pipeline replication to peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 12:17:16 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:17:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:19 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:17:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:20 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:17:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:21 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500253498s
2021/04/17 12:17:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:38554->10.242.244.80:8302: i/o timeout
2021/04/17 12:17:22 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:17:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:24 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.956032994s
2021/04/17 12:17:24 [ERR] memberlist: failed to receive: read tcp 10.242.253.198:8301->10.242.244.80:60184: i/o timeout from=10.242.244.80:60184
2021/04/17 12:17:24 [ERR] memberlist: Failed to send error: write tcp 10.242.253.198:8301->10.242.244.80:60184: i/o timeout from=10.242.244.80:60184
2021/04/17 12:17:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:26 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:48086->10.242.244.80:8300: i/o timeout
2021/04/17 12:17:26 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:17:26 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.411282146s
2021/04/17 12:17:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:28 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:17:28 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:17:30 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:17:31 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.503301043s
2021/04/17 12:17:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:38736->10.242.244.80:8302: i/o timeout
2021/04/17 12:17:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:17:32 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:17:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:34 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.933978013s
2021/04/17 12:17:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:36 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:48304->10.242.244.80:8300: i/o timeout
2021/04/17 12:17:36 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:17:37 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:17:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:38 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500487226s
2021/04/17 12:17:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:41 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.96994061s
2021/04/17 12:17:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:38926->10.242.244.80:8302: i/o timeout
2021/04/17 12:17:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:17:42 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:17:42 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:17:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:43 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.398309208s
2021/04/17 12:17:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:46 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:47332->10.242.244.80:8300: i/o timeout
2021/04/17 12:17:46 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:17:47 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:48522->10.242.244.80:8300: i/o timeout
2021/04/17 12:17:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:47 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:17:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:53 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:17:53 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:55434
2021/04/17 12:17:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:56 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:48678->10.242.244.80:8300: i/o timeout
2021/04/17 12:17:56 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:17:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:17:57 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:48710->10.242.244.80:8300: i/o timeout
2021/04/17 12:17:58 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:17:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:06 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:48880->10.242.244.80:8300: i/o timeout
2021/04/17 12:18:06 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:18:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:07 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:18:07 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:18:08 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:48928->10.242.244.80:8300: i/o timeout
2021/04/17 12:18:09 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:18:10 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:18:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:16 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:49080->10.242.244.80:8300: i/o timeout
2021/04/17 12:18:16 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:18:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:19 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:49138->10.242.244.80:8300: i/o timeout
2021/04/17 12:18:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:19 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:18:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:26 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:49262->10.242.244.80:8300: i/o timeout
2021/04/17 12:18:27 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:18:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:29 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:49360->10.242.244.80:8300: i/o timeout
2021/04/17 12:18:30 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:18:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:37 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:49470->10.242.244.80:8300: i/o timeout
2021/04/17 12:18:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:37 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:18:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:40 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:49530->10.242.244.80:8300: i/o timeout
2021/04/17 12:18:41 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:18:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:47 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:49694->10.242.244.80:8300: i/o timeout
2021/04/17 12:18:49 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:18:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:49 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:18:49 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:18:51 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:49750->10.242.244.80:8300: i/o timeout
2021/04/17 12:18:52 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:18:52 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:18:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:54 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:18:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:18:59 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:49922->10.242.244.80:8300: i/o timeout
2021/04/17 12:18:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:01 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:19:02 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:49976->10.242.244.80:8300: i/o timeout
2021/04/17 12:19:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:03 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:19:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:11 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:50134->10.242.244.80:8300: i/o timeout
2021/04/17 12:19:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:13 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:50192->10.242.244.80:8300: i/o timeout
2021/04/17 12:19:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:15 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:19:16 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:19:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:25 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:50400->10.242.244.80:8300: i/o timeout
2021/04/17 12:19:26 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:50414->10.242.244.80:8300: i/o timeout
2021/04/17 12:19:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:29 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:19:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:31 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:19:31 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:19:34 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:19:34 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 12:19:34 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 12:19:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:37 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:19:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:39 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:50678->10.242.244.80:8300: i/o timeout
2021/04/17 12:19:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:45 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:19:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:47 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:50834->10.242.244.80:8300: i/o timeout
2021/04/17 12:19:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:55 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:50978->10.242.244.80:8300: i/o timeout
2021/04/17 12:19:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:57 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:19:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:19:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:20:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:20:01 [ERR] memberlist: failed to receive: read tcp 10.242.253.198:8302->10.242.244.80:53810: i/o timeout from=10.242.244.80:53810
2021/04/17 12:20:01 [ERR] memberlist: Failed to send error: write tcp 10.242.253.198:8302->10.242.244.80:53810: i/o timeout from=10.242.244.80:53810
2021/04/17 12:20:02 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 12:20:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:20:04 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:20:04 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:20:04 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:20:05 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:20:05 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:20:05 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 12:20:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:20:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:56500->10.242.244.80:8301: i/o timeout
2021/04/17 12:20:06 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:20:06 [ERR] raft: peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300} has newer term, stopping replication
2021/04/17 12:20:06 [INFO] raft: Node at 10.242.253.198:8300 [Follower] entering Follower state (Leader: "")
2021/04/17 12:20:06 [INFO] raft: aborting pipeline replication to peer {Voter 38d8f917-7a55-e6fb-c4e2-f154d8409d0c 10.242.146.73:8300}
2021/04/17 12:20:06 [INFO] consul: cluster leadership lost
2021/04/17 12:20:08 [WARN] consul.coordinate: Batch update failed: node is not the leader
2021/04/17 12:20:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:56598->10.242.244.80:8301: i/o timeout
2021/04/17 12:20:10 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:20:10 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 12:20:10 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:20:10 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:20:16 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/17 12:20:16 [INFO] raft: Node at 10.242.253.198:8300 [Candidate] entering Candidate state in term 25
2021/04/17 12:20:16 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:20:16 [INFO] raft: Election won. Tally: 2
2021/04/17 12:20:16 [INFO] raft: Node at 10.242.253.198:8300 [Leader] entering Leader state
2021/04/17 12:20:16 [INFO] raft: Added peer 38d8f917-7a55-e6fb-c4e2-f154d8409d0c, starting replication
2021/04/17 12:20:16 [INFO] raft: Added peer a06ee723-0a7a-bed8-b418-2a3a48a30906, starting replication
2021/04/17 12:20:16 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:20:16 [INFO] consul: cluster leadership acquired
2021/04/17 12:20:16 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/17 12:20:16 [INFO] raft: pipelining replication to peer {Voter 38d8f917-7a55-e6fb-c4e2-f154d8409d0c 10.242.146.73:8300}
2021/04/17 12:20:16 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 12:20:16 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:20:18 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.502978976s
2021/04/17 12:20:19 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:20:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:20:20 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 12:20:21 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.929198135s
2021/04/17 12:20:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:20:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:42036->10.242.244.80:8302: i/o timeout
2021/04/17 12:20:22 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:20:23 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.388558775s
2021/04/17 12:20:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:20:26 [ERR] raft: Failed to make RequestVote RPC to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:51230->10.242.244.80:8300: i/o timeout
2021/04/17 12:20:26 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:51410->10.242.244.80:8300: i/o timeout
2021/04/17 12:20:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:20:26 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:20:26 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:51590->10.242.244.80:8300: i/o timeout
2021/04/17 12:20:27 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:20:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:20:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:20:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:20:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:42228->10.242.244.80:8302: i/o timeout
2021/04/17 12:20:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:20:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:20:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:20:36 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:51798->10.242.244.80:8300: i/o timeout
2021/04/17 12:20:36 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:20:37 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:51832->10.242.244.80:8300: i/o timeout
2021/04/17 12:20:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:20:38 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:20:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:20:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:20:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:20:45 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 12:20:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:20:46 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:51992->10.242.244.80:8300: i/o timeout
2021/04/17 12:20:46 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:20:46 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:20:46 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:20:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.253.198:42444->10.242.244.80:8302: i/o timeout
2021/04/17 12:20:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:20:47 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 12:20:47 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:20:47 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:20:48 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:52060->10.242.244.80:8300: i/o timeout
2021/04/17 12:20:49 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:20:49 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:20:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:20:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:20:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:20:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:20:56 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:52174->10.242.244.80:8300: i/o timeout
2021/04/17 12:20:56 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:20:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:20:59 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:52268->10.242.244.80:8300: i/o timeout
2021/04/17 12:21:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:00 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:21:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:06 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:52396->10.242.244.80:8300: i/o timeout
2021/04/17 12:21:06 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:21:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:10 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:52476->10.242.244.80:8300: i/o timeout
2021/04/17 12:21:11 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:21:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:16 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:52592->10.242.244.80:8300: i/o timeout
2021/04/17 12:21:16 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:21:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:21 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:52680->10.242.244.80:8300: i/o timeout
2021/04/17 12:21:22 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:21:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:26 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:52766->10.242.244.80:8300: i/o timeout
2021/04/17 12:21:26 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:21:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:28 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:21:28 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:21:31 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:21:32 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:52902->10.242.244.80:8300: i/o timeout
2021/04/17 12:21:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:33 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:21:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:36 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:52972->10.242.244.80:8300: i/o timeout
2021/04/17 12:21:37 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:21:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:41 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:21:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:43 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:53112->10.242.244.80:8300: i/o timeout
2021/04/17 12:21:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:44 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:21:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:47 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:53202->10.242.244.80:8300: i/o timeout
2021/04/17 12:21:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:48 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:21:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:54 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:53312->10.242.244.80:8300: i/o timeout
2021/04/17 12:21:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:56 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:21:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:21:58 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.253.198:53422->10.242.244.80:8300: i/o timeout
2021/04/17 12:22:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:22:01 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:22:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:22:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:22:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:22:06 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:22:06 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.253.198:53554->10.242.244.80:8300: i/o timeout
2021/04/17