By type
+ CONSUL_BIND=
+ [ -n ]
+ CONSUL_CLIENT=
+ [ -n ]
+ CONSUL_DATA_DIR=/consul/data
+ CONSUL_CONFIG_DIR=/consul/config
+ [ -n ]
+ [ a = - ]
+ [ agent = agent ]
+ shift
+ set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
+ [ consul = consul ]
+ stat -c %u /consul/data
+ id -u consul
+ [ 100 != 100 ]
+ stat -c %u /consul/config
+ id -u consul
+ [ 100 != 100 ]
+ [ ! -z ]
+ exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
==> Starting Consul agent...
bootstrap_expect > 0: expecting 3 servers
==> Consul agent running!
Version: 'v1.0.6'
Node ID: 'bd107711-20be-aec5-53f2-f124e65e537b'
Node name: 'dev-consul-server-0'
Datacenter: 'dc1' (Segment: '')
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.242.240.138 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2021/04/12 22:40:37 [INFO] raft: Initial configuration (index=0): []
2021/04/12 22:40:37 [INFO] raft: Node at 10.242.240.138:8300 [Follower] entering Follower state (Leader: "")
2021/04/12 22:40:37 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.240.138
2021/04/12 22:40:37 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.240.138
2021/04/12 22:40:37 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/04/12 22:40:37 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2021/04/12 22:40:37 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.240.138:8300) (DC: dc1)
2021/04/12 22:40:37 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2021/04/12 22:40:37 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
2021/04/12 22:40:37 [INFO] agent: started state syncer
2021/04/12 22:40:37 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer
2021/04/12 22:40:37 [INFO] agent: Joining LAN cluster...
2021/04/12 22:40:37 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/04/12 22:40:37 [WARN] memberlist: Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
2021/04/12 22:40:37 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
2021/04/12 22:40:37 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/04/12 22:40:37 [INFO] agent: (LAN) joined: 0 Err: 3 error(s) occurred:
* Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/04/12 22:40:37 [WARN] agent: Join LAN failed: , retrying in 30s
2021/04/12 22:40:42 [WARN] raft: no known peers, aborting election
2021/04/12 22:40:44 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/12 22:40:52 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.149.73
2021/04/12 22:40:52 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.149.73:8300) (DC: dc1)
2021/04/12 22:40:52 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.149.73
2021/04/12 22:40:52 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/12 22:41:03 [ERR] agent: Coordinate update error: No cluster leader
2021/04/12 22:41:06 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.53.8
2021/04/12 22:41:06 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.53.8:8300) (DC: dc1)
2021/04/12 22:41:06 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.242.240.138:8300,10.242.149.73:8300,10.242.53.8:8300
2021/04/12 22:41:06 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.53.8
2021/04/12 22:41:06 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/12 22:41:06 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/12 22:41:06 [INFO] raft: Node at 10.242.240.138:8300 [Candidate] entering Candidate state in term 2
2021/04/12 22:41:06 [INFO] raft: Election won. Tally: 2
2021/04/12 22:41:06 [INFO] raft: Node at 10.242.240.138:8300 [Leader] entering Leader state
2021/04/12 22:41:06 [INFO] raft: Added peer 59c594ab-4dd4-45c8-f46f-385adf64f752, starting replication
2021/04/12 22:41:06 [INFO] raft: Added peer f24a3ca2-ae0f-60e3-7606-21329dbc0417, starting replication
2021/04/12 22:41:06 [INFO] consul: cluster leadership acquired
2021/04/12 22:41:06 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/12 22:41:06 [INFO] raft: pipelining replication to peer {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}
2021/04/12 22:41:06 [WARN] raft: AppendEntries to {Voter 59c594ab-4dd4-45c8-f46f-385adf64f752 10.242.149.73:8300} rejected, sending older logs (next: 1)
2021/04/12 22:41:06 [INFO] raft: pipelining replication to peer {Voter 59c594ab-4dd4-45c8-f46f-385adf64f752 10.242.149.73:8300}
2021/04/12 22:41:06 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/04/12 22:41:07 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/12 22:41:07 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/12 22:41:07 [INFO] agent: Synced node info
2021/04/12 22:41:07 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/04/12 22:41:07 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/04/12 22:41:07 [INFO] agent: (LAN) joined: 2 Err:
2021/04/12 22:41:07 [INFO] agent: Join LAN completed. Synced with 2 initial agents
==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=79959c58-430b-bd23-7007-6b8b6c07c83a&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2021/04/12 22:41:17 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-qtrtq 10.242.240.139
2021/04/12 22:41:17 [INFO] consul: member 'dev-consul-7467bf7b46-qtrtq' joined, marking health alive
2021/04/12 22:42:13 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/12 22:42:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/12 22:58:43 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/12 22:58:43 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/12 22:58:43 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/12 23:01:29 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/12 23:01:29 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/12 23:01:52 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/12 23:01:52 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/12 23:01:52 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:02:03 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/12 23:02:03 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/12 23:02:03 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:02:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/12 23:05:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.240.138:51468->10.242.53.8:8301: i/o timeout
2021/04/12 23:05:03 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/12 23:05:05 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:05:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.240.138:51562->10.242.53.8:8301: i/o timeout
2021/04/12 23:05:06 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/12 23:05:06 [WARN] raft: Failed to contact f24a3ca2-ae0f-60e3-7606-21329dbc0417 in 2.500253103s
2021/04/12 23:05:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.240.138:53122->10.242.53.8:8302: i/o timeout
2021/04/12 23:05:07 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/12 23:05:07 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/12 23:05:07 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.53.8
2021/04/12 23:05:07 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.53.8:8300) (DC: dc1)
2021/04/12 23:05:07 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/12 23:05:08 [WARN] raft: Failed to contact f24a3ca2-ae0f-60e3-7606-21329dbc0417 in 4.920419347s
2021/04/12 23:05:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:11 [WARN] raft: Failed to contact f24a3ca2-ae0f-60e3-7606-21329dbc0417 in 7.37375821s
2021/04/12 23:05:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:14 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:40838->10.242.53.8:8300: i/o timeout
2021/04/12 23:05:14 [INFO] raft: aborting pipeline replication to peer {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}
2021/04/12 23:05:14 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:05:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:15 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:05:16 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/12 23:05:16 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/12 23:05:16 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.53.8:37896
2021/04/12 23:05:16 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/12 23:05:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.240.138:53296->10.242.53.8:8302: i/o timeout
2021/04/12 23:05:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/12 23:05:17 [WARN] raft: Failed to contact f24a3ca2-ae0f-60e3-7606-21329dbc0417 in 3.712495119s
2021/04/12 23:05:18 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:05:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:20 [WARN] raft: Failed to contact f24a3ca2-ae0f-60e3-7606-21329dbc0417 in 6.206970003s
2021/04/12 23:05:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:24 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:33044->10.242.53.8:8300: i/o timeout
2021/04/12 23:05:24 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:05:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:25 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:33052->10.242.53.8:8300: i/o timeout
2021/04/12 23:05:25 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:05:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.240.138:53608->10.242.53.8:8302: i/o timeout
2021/04/12 23:05:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/12 23:05:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:34 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:33248->10.242.53.8:8300: i/o timeout
2021/04/12 23:05:34 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:05:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:35 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:33282->10.242.53.8:8300: i/o timeout
2021/04/12 23:05:36 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:05:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:37 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/12 23:05:37 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.53.8
2021/04/12 23:05:37 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/12 23:05:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:44 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:33456->10.242.53.8:8300: i/o timeout
2021/04/12 23:05:44 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:05:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:46 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:33516->10.242.53.8:8300: i/o timeout
2021/04/12 23:05:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:47 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:05:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:54 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:33658->10.242.53.8:8300: i/o timeout
2021/04/12 23:05:54 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:05:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:05:57 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:33718->10.242.53.8:8300: i/o timeout
2021/04/12 23:05:57 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:05:57 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/12 23:05:57 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/12 23:06:02 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:06:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:04 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:33888->10.242.53.8:8300: i/o timeout
2021/04/12 23:06:04 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:06:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:07 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.53.8:8301
2021/04/12 23:06:07 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.53.8:8302
2021/04/12 23:06:07 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:33954->10.242.53.8:8300: i/o timeout
2021/04/12 23:06:08 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:06:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:14 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:34052->10.242.53.8:8300: i/o timeout
2021/04/12 23:06:14 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:06:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:18 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:34212->10.242.53.8:8300: i/o timeout
2021/04/12 23:06:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:19 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:06:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:24 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:34314->10.242.53.8:8300: i/o timeout
2021/04/12 23:06:24 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:06:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:29 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:34408->10.242.53.8:8300: i/o timeout
2021/04/12 23:06:30 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:06:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:34 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:34532->10.242.53.8:8300: i/o timeout
2021/04/12 23:06:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:35 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:06:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:40 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:34656->10.242.53.8:8300: i/o timeout
2021/04/12 23:06:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:41 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/12 23:06:41 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/12 23:06:42 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:06:44 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:06:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:45 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:34766->10.242.53.8:8300: i/o timeout
2021/04/12 23:06:46 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:06:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:47 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.53.8:8301
2021/04/12 23:06:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:52 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:34904->10.242.53.8:8300: i/o timeout
2021/04/12 23:06:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:56 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:06:56 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:34962->10.242.53.8:8300: i/o timeout
2021/04/12 23:06:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:06:59 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:07:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:06 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:35162->10.242.53.8:8300: i/o timeout
2021/04/12 23:07:07 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.53.8:8302
2021/04/12 23:07:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:09 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:35214->10.242.53.8:8300: i/o timeout
2021/04/12 23:07:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:12 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:07:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:14 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:07:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:22 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:35500->10.242.53.8:8300: i/o timeout
2021/04/12 23:07:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:23 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/12 23:07:23 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/12 23:07:24 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:35560->10.242.53.8:8300: i/o timeout
2021/04/12 23:07:25 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:07:25 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/12 23:07:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:33 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:07:34 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:07:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:43 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:35926->10.242.53.8:8300: i/o timeout
2021/04/12 23:07:44 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:35956->10.242.53.8:8300: i/o timeout
2021/04/12 23:07:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:53 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:07:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:55 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:07:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:07:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:03 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:36324->10.242.53.8:8300: i/o timeout
2021/04/12 23:08:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:04 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/12 23:08:04 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/12 23:08:05 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:36342->10.242.53.8:8300: i/o timeout
2021/04/12 23:08:07 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:08:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:14 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:08:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:15 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:08:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:24 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:36772->10.242.53.8:8300: i/o timeout
2021/04/12 23:08:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:25 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:36782->10.242.53.8:8300: i/o timeout
2021/04/12 23:08:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:35 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:08:35 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:08:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:37 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.53.8:8302
2021/04/12 23:08:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:45 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:37170->10.242.53.8:8300: i/o timeout
2021/04/12 23:08:45 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:37178->10.242.53.8:8300: i/o timeout
2021/04/12 23:08:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:46 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/12 23:08:46 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/12 23:08:49 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:08:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:55 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:08:56 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:08:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:08:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:05 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:37552->10.242.53.8:8300: i/o timeout
2021/04/12 23:09:06 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:37570->10.242.53.8:8300: i/o timeout
2021/04/12 23:09:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:15 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:09:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:17 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:09:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:25 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:37978->10.242.53.8:8300: i/o timeout
2021/04/12 23:09:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:27 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:38016->10.242.53.8:8300: i/o timeout
2021/04/12 23:09:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:28 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/12 23:09:28 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/12 23:09:31 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:09:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:36 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:09:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:37 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.53.8:8302
2021/04/12 23:09:38 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:09:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:46 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:38388->10.242.53.8:8300: i/o timeout
2021/04/12 23:09:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:48 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:38428->10.242.53.8:8300: i/o timeout
2021/04/12 23:09:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:56 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:09:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:09:59 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:10:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:06 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:38766->10.242.53.8:8300: i/o timeout
2021/04/12 23:10:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:09 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:38824->10.242.53.8:8300: i/o timeout
2021/04/12 23:10:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:10 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/12 23:10:10 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/12 23:10:13 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:10:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:16 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:10:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:20 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:10:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:26 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:39190->10.242.53.8:8300: i/o timeout
2021/04/12 23:10:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:30 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:39266->10.242.53.8:8300: i/o timeout
2021/04/12 23:10:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:36 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:10:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:37 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.53.8:8302
2021/04/12 23:10:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:41 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:10:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:46 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:39594->10.242.53.8:8300: i/o timeout
2021/04/12 23:10:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:51 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:39676->10.242.53.8:8300: i/o timeout
2021/04/12 23:10:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:52 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/12 23:10:52 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/12 23:10:55 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:10:55 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/12 23:10:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:10:57 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:10:57 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.53.8:8301
2021/04/12 23:10:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:02 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:11:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:07 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:40010->10.242.53.8:8300: i/o timeout
2021/04/12 23:11:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:12 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:40134->10.242.53.8:8300: i/o timeout
2021/04/12 23:11:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:17 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:11:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:23 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:11:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:27 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:40472->10.242.53.8:8300: i/o timeout
2021/04/12 23:11:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:33 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:40574->10.242.53.8:8300: i/o timeout
2021/04/12 23:11:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:34 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/12 23:11:34 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/12 23:11:37 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.53.8:8302
2021/04/12 23:11:37 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:11:37 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:11:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:44 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:11:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:47 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:40880->10.242.53.8:8300: i/o timeout
2021/04/12 23:11:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:54 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:41008->10.242.53.8:8300: i/o timeout
2021/04/12 23:11:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:11:57 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:11:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:05 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:12:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:07 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:41280->10.242.53.8:8300: i/o timeout
2021/04/12 23:12:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:15 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:41458->10.242.53.8:8300: i/o timeout
2021/04/12 23:12:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:16 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/12 23:12:16 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/12 23:12:18 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:12:19 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:12:19 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/12 23:12:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:26 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:12:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:28 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:41796->10.242.53.8:8300: i/o timeout
2021/04/12 23:12:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:36 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:42128->10.242.53.8:8300: i/o timeout
2021/04/12 23:12:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:37 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.53.8:8301
2021/04/12 23:12:37 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.53.8:8302
2021/04/12 23:12:38 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:12:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:48 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:12:48 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:42362->10.242.53.8:8300: i/o timeout
2021/04/12 23:12:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:58 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:42580->10.242.53.8:8300: i/o timeout
2021/04/12 23:12:58 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:12:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:12:58 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/12 23:12:58 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/12 23:13:02 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:13:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:08 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:42790->10.242.53.8:8300: i/o timeout
2021/04/12 23:13:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:09 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:13:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:18 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:13:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:19 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:43048->10.242.53.8:8300: i/o timeout
2021/04/12 23:13:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:28 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:43288->10.242.53.8:8300: i/o timeout
2021/04/12 23:13:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:29 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:13:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:37 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.53.8:8302
2021/04/12 23:13:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:39 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:13:39 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:43552->10.242.53.8:8300: i/o timeout
2021/04/12 23:13:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:41 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/12 23:13:41 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/12 23:13:43 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:13:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:49 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:43742->10.242.53.8:8300: i/o timeout
2021/04/12 23:13:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:51 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:13:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:13:59 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:14:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:01 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:43978->10.242.53.8:8300: i/o timeout
2021/04/12 23:14:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:09 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:44180->10.242.53.8:8300: i/o timeout
2021/04/12 23:14:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:12 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:14:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:19 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:14:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:22 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:44510->10.242.53.8:8300: i/o timeout
2021/04/12 23:14:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:22 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/12 23:14:22 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/12 23:14:25 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:14:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:29 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:44634->10.242.53.8:8300: i/o timeout
2021/04/12 23:14:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:33 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:14:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:37 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.53.8:8302
2021/04/12 23:14:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:39 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:14:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:43 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:44938->10.242.53.8:8300: i/o timeout
2021/04/12 23:14:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:49 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:45100->10.242.53.8:8300: i/o timeout
2021/04/12 23:14:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:54 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:14:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:14:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:00 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:15:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:04 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:45424->10.242.53.8:8300: i/o timeout
2021/04/12 23:15:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:04 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/12 23:15:04 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/12 23:15:07 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:15:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:10 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.53.8
2021/04/12 23:15:10 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/12 23:15:10 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:45564->10.242.53.8:8300: i/o timeout
2021/04/12 23:15:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:15 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:15:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:20 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:15:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:25 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:45930->10.242.53.8:8300: i/o timeout
2021/04/12 23:15:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.240.138:38162->10.242.53.8:8302: i/o timeout
2021/04/12 23:15:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/12 23:15:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:30 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:46054->10.242.53.8:8300: i/o timeout
2021/04/12 23:15:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:36 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:15:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:40 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:15:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.240.138:38380->10.242.53.8:8302: i/o timeout
2021/04/12 23:15:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/12 23:15:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:46 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:46404->10.242.53.8:8300: i/o timeout
2021/04/12 23:15:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:46 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/12 23:15:46 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/12 23:15:47 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/12 23:15:47 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.53.8
2021/04/12 23:15:47 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/12 23:15:47 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.53.8:8301
2021/04/12 23:15:49 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:15:50 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:46510->10.242.53.8:8300: i/o timeout
2021/04/12 23:15:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:56 [ERR] memberlist: failed to receive: read tcp 10.242.240.138:8302->10.242.53.8:45972: i/o timeout from=10.242.53.8:45972
2021/04/12 23:15:56 [ERR] memberlist: Failed to send error: write tcp 10.242.240.138:8302->10.242.53.8:45972: i/o timeout from=10.242.53.8:45972
2021/04/12 23:15:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:15:57 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:15:57 [DEBUG] raft-net: 10.242.240.138:8300 accepted connection from: 10.242.53.8:35344
2021/04/12 23:15:57 [WARN] raft: Rejecting vote request from 10.242.53.8:8300 since we have a leader: 10.242.240.138:8300
2021/04/12 23:15:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:16:00 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:16:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:16:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:16:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:16:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:16:07 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:46866->10.242.53.8:8300: i/o timeout
2021/04/12 23:16:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:16:10 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:46944->10.242.53.8:8300: i/o timeout
2021/04/12 23:16:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:16:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:16:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:16:16 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.53.8
2021/04/12 23:16:16 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/12 23:16:16 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.53.8
2021/04/12 23:16:16 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.53.8:8300) (DC: dc1)
2021/04/12 23:16:16 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/12 23:16:21 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:16:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:16:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.240.138:37906->10.242.53.8:8301: i/o timeout
2021/04/12 23:16:24 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/12 23:16:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:16:25 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/12 23:16:25 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.53.8
2021/04/12 23:16:25 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.53.8:8300) (DC: dc1)
2021/04/12 23:16:25 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/12 23:16:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:16:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.240.138:39514->10.242.53.8:8302: i/o timeout
2021/04/12 23:16:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/12 23:16:28 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: read tcp 10.242.240.138:47360->10.242.53.8:8300: i/o timeout
2021/04/12 23:16:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:16:29 [ERR] raft: peer {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300} has newer term, stopping replication
2021/04/12 23:16:29 [INFO] raft: Node at 10.242.240.138:8300 [Follower] entering Follower state (Leader: "")
2021/04/12 23:16:29 [INFO] raft: aborting pipeline replication to peer {Voter 59c594ab-4dd4-45c8-f46f-385adf64f752 10.242.149.73:8300}
2021/04/12 23:16:29 [INFO] consul: cluster leadership lost
2021/04/12 23:16:35 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.53.8
2021/04/12 23:16:35 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.53.8:8300) (DC: dc1)
2021/04/12 23:16:35 [WARN] raft: Rejecting vote request from 10.242.53.8:8300 since our last index is greater (473, 356)
2021/04/12 23:16:37 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/12 23:16:37 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/12 23:16:37 [INFO] raft: Node at 10.242.240.138:8300 [Candidate] entering Candidate state in term 5
2021/04/12 23:16:37 [INFO] raft: Election won. Tally: 2
2021/04/12 23:16:37 [INFO] raft: Node at 10.242.240.138:8300 [Leader] entering Leader state
2021/04/12 23:16:37 [INFO] raft: Added peer 59c594ab-4dd4-45c8-f46f-385adf64f752, starting replication
2021/04/12 23:16:37 [INFO] raft: Added peer f24a3ca2-ae0f-60e3-7606-21329dbc0417, starting replication
2021/04/12 23:16:37 [INFO] consul: cluster leadership acquired
2021/04/12 23:16:37 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/12 23:16:37 [INFO] raft: pipelining replication to peer {Voter 59c594ab-4dd4-45c8-f46f-385adf64f752 10.242.149.73:8300}
2021/04/12 23:16:37 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/12 23:16:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.240.138:38226->10.242.53.8:8301: i/o timeout
2021/04/12 23:16:38 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/12 23:16:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:16:40 [WARN] raft: Failed to contact f24a3ca2-ae0f-60e3-7606-21329dbc0417 in 2.512475097s
2021/04/12 23:16:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.240.138:38306->10.242.53.8:8301: i/o timeout
2021/04/12 23:16:41 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/12 23:16:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:16:42 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.53.8
2021/04/12 23:16:42 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.53.8:8300) (DC: dc1)
2021/04/12 23:16:42 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/12 23:16:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.240.138:39864->10.242.53.8:8302: i/o timeout
2021/04/12 23:16:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/12 23:16:42 [WARN] raft: Failed to contact f24a3ca2-ae0f-60e3-7606-21329dbc0417 in 4.959564495s
2021/04/12 23:16:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:16:45 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.53.8
2021/04/12 23:16:45 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.53.8:8300) (DC: dc1)
2021/04/12 23:16:45 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/12 23:16:45 [WARN] raft: Failed to contact f24a3ca2-ae0f-60e3-7606-21329dbc0417 in 7.427664039s
2021/04/12 23:16:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:16:46 [WARN] raft: Rejecting vote request from 10.242.53.8:8300 since we have a leader: 10.242.240.138:8300
2021/04/12 23:16:46 [WARN] raft: AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300} rejected, sending older logs (next: 357)
2021/04/12 23:16:48 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:16:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:16:51 [WARN] raft: Failed to contact f24a3ca2-ae0f-60e3-7606-21329dbc0417 in 2.50028781s
2021/04/12 23:16:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:16:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.240.138:38550->10.242.53.8:8301: i/o timeout
2021/04/12 23:16:52 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/12 23:16:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:16:54 [WARN] raft: Failed to contact f24a3ca2-ae0f-60e3-7606-21329dbc0417 in 4.976851401s
2021/04/12 23:16:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.240.138:38598->10.242.53.8:8301: i/o timeout
2021/04/12 23:16:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/12 23:16:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:16:56 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/12 23:16:56 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.53.8
2021/04/12 23:16:56 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.53.8:8300) (DC: dc1)
2021/04/12 23:16:56 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/12 23:16:56 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.53.8
2021/04/12 23:16:56 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.53.8:8300) (DC: dc1)
2021/04/12 23:16:56 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/12 23:16:57 [INFO] raft: pipelining replication to peer {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}
2021/04/12 23:17:06 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: EOF
2021/04/12 23:17:06 [INFO] raft: aborting pipeline replication to peer {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}
2021/04/12 23:17:06 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: read tcp 10.242.240.138:47810->10.242.53.8:8300: read: connection reset by peer
2021/04/12 23:17:06 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: dial tcp ->10.242.53.8:8300: getsockopt: connection refused
2021/04/12 23:17:06 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: dial tcp ->10.242.53.8:8300: getsockopt: connection refused
2021/04/12 23:17:06 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: dial tcp ->10.242.53.8:8300: getsockopt: connection refused
2021/04/12 23:17:06 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: dial tcp ->10.242.53.8:8300: getsockopt: connection refused
2021/04/12 23:17:07 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: dial tcp ->10.242.53.8:8300: getsockopt: connection refused
2021/04/12 23:17:07 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: dial tcp ->10.242.53.8:8300: getsockopt: connection refused
2021/04/12 23:17:07 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: dial tcp ->10.242.53.8:8300: getsockopt: connection refused
2021/04/12 23:17:07 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: dial tcp ->10.242.53.8:8300: getsockopt: connection refused
2021/04/12 23:17:07 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: stream closed
2021/04/12 23:17:08 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: dial tcp ->10.242.53.8:8300: getsockopt: connection refused
2021/04/12 23:17:08 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/12 23:17:08 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: dial tcp ->10.242.53.8:8300: getsockopt: connection refused
2021/04/12 23:17:08 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: dial tcp ->10.242.53.8:8300: getsockopt: connection refused
2021/04/12 23:17:08 [WARN] raft: Failed to contact f24a3ca2-ae0f-60e3-7606-21329dbc0417 in 2.502246044s
2021/04/12 23:17:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:17:08 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/12 23:17:09 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: dial tcp ->10.242.53.8:8300: getsockopt: connection refused
2021/04/12 23:17:09 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: dial tcp ->10.242.53.8:8300: getsockopt: connection refused
2021/04/12 23:17:09 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.53.8:8300: getsockopt: connection refused
2021/04/12 23:17:10 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: dial tcp ->10.242.53.8:8300: getsockopt: connection refused
2021/04/12 23:17:10 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:17:11 [WARN] raft: Failed to contact f24a3ca2-ae0f-60e3-7606-21329dbc0417 in 4.995361084s
2021/04/12 23:17:11 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/12 23:17:12 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/12 23:17:12 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/12 23:17:12 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.53.8
2021/04/12 23:17:12 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.53.8:8300) (DC: dc1)
2021/04/12 23:17:12 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/12 23:17:12 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:17:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:17:13 [WARN] raft: Failed to contact f24a3ca2-ae0f-60e3-7606-21329dbc0417 in 7.453279126s
2021/04/12 23:17:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:17:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:17:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:17:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:17:21 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:17:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:17:21 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:17:22 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:17:22 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:17:24 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:17:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:17:27 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:17:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:17:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:17:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:17:32 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:17:33 [ERR] memberlist: Conflicting address for dev-consul-server-2. Mine: 10.242.53.8:8301 Theirs: 10.242.53.46:8301
2021/04/12 23:17:33 [WARN] serf: Name conflict for 'dev-consul-server-2' both 10.242.53.8:8301 and 10.242.53.46:8301 are claiming
2021/04/12 23:17:33 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.53.8:8302 Theirs: 10.242.53.46:8302
2021/04/12 23:17:33 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.53.8:8302 and 10.242.53.46:8302 are claiming
2021/04/12 23:17:33 [ERR] memberlist: Conflicting address for dev-consul-server-2. Mine: 10.242.53.8:8301 Theirs: 10.242.53.46:8301
2021/04/12 23:17:33 [WARN] serf: Name conflict for 'dev-consul-server-2' both 10.242.53.8:8301 and 10.242.53.46:8301 are claiming
2021/04/12 23:17:33 [ERR] memberlist: Conflicting address for dev-consul-server-2. Mine: 10.242.53.8:8301 Theirs: 10.242.53.46:8301
2021/04/12 23:17:33 [WARN] serf: Name conflict for 'dev-consul-server-2' both 10.242.53.8:8301 and 10.242.53.46:8301 are claiming
2021/04/12 23:17:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:17:33 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:17:33 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:17:33 [ERR] memberlist: Conflicting address for dev-consul-server-2. Mine: 10.242.53.8:8301 Theirs: 10.242.53.46:8301
2021/04/12 23:17:33 [WARN] serf: Name conflict for 'dev-consul-server-2' both 10.242.53.8:8301 and 10.242.53.46:8301 are claiming
2021/04/12 23:17:33 [ERR] memberlist: Conflicting address for dev-consul-server-2. Mine: 10.242.53.8:8301 Theirs: 10.242.53.46:8301
2021/04/12 23:17:33 [WARN] serf: Name conflict for 'dev-consul-server-2' both 10.242.53.8:8301 and 10.242.53.46:8301 are claiming
2021/04/12 23:17:34 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.53.8:8302 Theirs: 10.242.53.46:8302
2021/04/12 23:17:34 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.53.8:8302 and 10.242.53.46:8302 are claiming
2021/04/12 23:17:34 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.53.8:8302 Theirs: 10.242.53.46:8302
2021/04/12 23:17:34 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.53.8:8302 and 10.242.53.46:8302 are claiming
2021/04/12 23:17:34 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.53.8:8302 Theirs: 10.242.53.46:8302
2021/04/12 23:17:34 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.53.8:8302 and 10.242.53.46:8302 are claiming
2021/04/12 23:17:34 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.53.8:8302 Theirs: 10.242.53.46:8302
2021/04/12 23:17:34 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.53.8:8302 and 10.242.53.46:8302 are claiming
2021/04/12 23:17:36 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:17:37 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:17:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:17:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:17:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:17:42 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/12 23:17:42 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.53.8
2021/04/12 23:17:42 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/12 23:17:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:17:43 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:17:45 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/12 23:17:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:17:45 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:17:46 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:17:47 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:17:48 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:17:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:17:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:17:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:17:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:17:56 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:17:57 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:17:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:17:57 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:17:59 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:18:00 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:18:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:18:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:18:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:18:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:18:08 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:18:09 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:18:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:18:09 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:18:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:18:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:18:15 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:18:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:18:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:18:18 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:18:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:18:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:18:21 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:18:24 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/12 23:18:25 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:18:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:18:27 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.53.8:8301
2021/04/12 23:18:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/12 23:18:28 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.53.46
2021/04/12 23:18:28 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.53.46:8300) (DC: dc1)
2021/04/12 23:18:28 [INFO] raft: Updating configuration with AddNonvoter (4c1d3206-1372-d166-23e9-0581483325c4, 10.242.53.46:8300) to [{Suffrage:Voter ID:bd107711-20be-aec5-53f2-f124e65e537b Address:10.242.240.138:8300} {Suffrage:Voter ID:59c594ab-4dd4-45c8-f46f-385adf64f752 Address:10.242.149.73:8300} {Suffrage:Voter ID:f24a3ca2-ae0f-60e3-7606-21329dbc0417 Address:10.242.53.8:8300} {Suffrage:Nonvoter ID:4c1d3206-1372-d166-23e9-0581483325c4 Address:10.242.53.46:8300}]
2021/04/12 23:18:28 [INFO] raft: Added peer 4c1d3206-1372-d166-23e9-0581483325c4, starting replication
2021/04/12 23:18:28 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/12 23:18:28 [WARN] raft: AppendEntries to {Nonvoter 4c1d3206-1372-d166-23e9-0581483325c4 10.242.53.46:8300} rejected, sending older logs (next: 1)
2021/04/12 23:18:28 [INFO] raft: pipelining replication to peer {Nonvoter 4c1d3206-1372-d166-23e9-0581483325c4 10.242.53.46:8300}
2021/04/12 23:18:28 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:18:33 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:18:36 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:18:38 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:18:39 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.53.8:8302 Theirs: 10.242.53.46:8302
2021/04/12 23:18:39 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.53.8:8302 and 10.242.53.46:8302 are claiming
2021/04/12 23:18:46 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:18:47 [INFO] autopilot: Promoting Server (ID: "4c1d3206-1372-d166-23e9-0581483325c4" Address: "10.242.53.46:8300") to voter
2021/04/12 23:18:47 [INFO] raft: Updating configuration with AddStaging (4c1d3206-1372-d166-23e9-0581483325c4, 10.242.53.46:8300) to [{Suffrage:Voter ID:bd107711-20be-aec5-53f2-f124e65e537b Address:10.242.240.138:8300} {Suffrage:Voter ID:59c594ab-4dd4-45c8-f46f-385adf64f752 Address:10.242.149.73:8300} {Suffrage:Voter ID:f24a3ca2-ae0f-60e3-7606-21329dbc0417 Address:10.242.53.8:8300} {Suffrage:Voter ID:4c1d3206-1372-d166-23e9-0581483325c4 Address:10.242.53.46:8300}]
2021/04/12 23:18:47 [INFO] autopilot: Attempting removal of stale Server (ID: "f24a3ca2-ae0f-60e3-7606-21329dbc0417" Address: "10.242.53.8:8300")
2021/04/12 23:18:47 [INFO] raft: Updating configuration with RemoveServer (f24a3ca2-ae0f-60e3-7606-21329dbc0417, ) to [{Suffrage:Voter ID:bd107711-20be-aec5-53f2-f124e65e537b Address:10.242.240.138:8300} {Suffrage:Voter ID:59c594ab-4dd4-45c8-f46f-385adf64f752 Address:10.242.149.73:8300} {Suffrage:Voter ID:4c1d3206-1372-d166-23e9-0581483325c4 Address:10.242.53.46:8300}]
2021/04/12 23:18:47 [INFO] raft: Removed peer f24a3ca2-ae0f-60e3-7606-21329dbc0417, stopping replication after 505
2021/04/12 23:18:48 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:18:57 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:18:58 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:19:07 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.53.8:8302
2021/04/12 23:19:07 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:19:08 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:19:18 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:19:18 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:19:28 [ERR] raft: Failed to heartbeat to 10.242.53.8:8300: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:19:29 [WARN] Unable to get address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417, using fallback address 10.242.53.8:8300: Could not find address for server id f24a3ca2-ae0f-60e3-7606-21329dbc0417
2021/04/12 23:19:39 [ERR] raft: Failed to AppendEntries to {Voter f24a3ca2-ae0f-60e3-7606-21329dbc0417 10.242.53.8:8300}: dial tcp ->10.242.53.8:8300: i/o timeout
2021/04/12 23:19:39 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.53.46
2021/04/12 23:19:39 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"