By type
+ CONSUL_BIND= + [ -n ] + CONSUL_CLIENT= + [ -n ] + CONSUL_DATA_DIR=/consul/data + CONSUL_CONFIG_DIR=/consul/config + [ -n ] + [ a = - ] + [ agent = agent ] + shift + set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui + [ consul = consul ] + stat -c %u /consul/data + id -u consul + [ 100 != 100 ] + stat -c %u /consul/config + id -u consul + [ 100 != 100 ] + [ ! -z ] + exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui bootstrap_expect > 0: expecting 3 servers ==> Starting Consul agent... ==> Consul agent running! Version: 'v1.0.6' Node ID: '8c9b3e16-04a3-1dc7-b2b3-e1ac04ac3526' Node name: 'dev-consul-server-0' Datacenter: 'dc1' (Segment: '') Server: true (Bootstrap: false) Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600) Cluster Addr: 10.242.29.201 (LAN: 8301, WAN: 8302) Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false ==> Log data will now stream in as it occurs: 2021/04/08 10:37:55 [INFO] raft: Initial configuration (index=0): [] 2021/04/08 10:37:55 [INFO] raft: Node at 10.242.29.201:8300 [Follower] entering Follower state (Leader: "") 2021/04/08 10:37:55 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.29.201 2021/04/08 10:37:55 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.29.201 2021/04/08 10:37:55 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.29.201:8300) (DC: dc1) 2021/04/08 10:37:55 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan" 2021/04/08 10:37:55 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp) 2021/04/08 10:37:55 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp) 2021/04/08 10:37:55 [INFO] agent: Started HTTP server on [::]:8500 (tcp) 2021/04/08 10:37:55 [INFO] agent: started state syncer 2021/04/08 10:37:55 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer 2021/04/08 10:37:55 [INFO] agent: Joining LAN cluster... 2021/04/08 10:37:55 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server] 2021/04/08 10:37:55 [WARN] memberlist: Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host 2021/04/08 10:37:56 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host 2021/04/08 10:37:56 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host 2021/04/08 10:37:56 [INFO] agent: (LAN) joined: 0 Err: 3 error(s) occurred: * Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host * Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host * Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host 2021/04/08 10:37:56 [WARN] agent: Join LAN failed: , retrying in 30s 2021/04/08 10:38:02 [WARN] raft: no known peers, aborting election 2021/04/08 10:38:02 [ERR] agent: failed to sync remote state: No cluster leader 2021/04/08 10:38:09 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-krcb7 10.242.94.71 ==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=b4a9f195-0ff5-3194-d00e-0c2d150c958d&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 2021/04/08 10:38:18 [ERR] agent: Coordinate update error: No cluster leader 2021/04/08 10:38:26 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server] 2021/04/08 10:38:26 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host 2021/04/08 10:38:26 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host 2021/04/08 10:38:26 [INFO] agent: (LAN) joined: 1 Err: 2021/04/08 10:38:26 [INFO] agent: Join LAN completed. Synced with 1 initial agents 2021/04/08 10:38:37 [ERR] agent: failed to sync remote state: No cluster leader 2021/04/08 10:38:49 [ERR] agent: Coordinate update error: No cluster leader 2021/04/08 10:39:12 [ERR] agent: failed to sync remote state: No cluster leader 2021/04/08 10:39:24 [ERR] agent: Coordinate update error: No cluster leader 2021/04/08 10:39:47 [ERR] agent: Coordinate update error: No cluster leader 2021/04/08 10:39:48 [ERR] agent: failed to sync remote state: No cluster leader 2021/04/08 10:39:56 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.53.14 2021/04/08 10:39:56 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.53.14:8300) (DC: dc1) 2021/04/08 10:39:56 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.53.14 2021/04/08 10:39:56 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan" 2021/04/08 10:40:13 [ERR] agent: failed to sync remote state: No cluster leader 2021/04/08 10:40:16 [ERR] agent: Coordinate update error: No cluster leader 2021/04/08 10:40:33 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.31.214 2021/04/08 10:40:33 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.31.214:8300) (DC: dc1) 2021/04/08 10:40:33 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.242.31.214:8300,10.242.29.201:8300,10.242.53.14:8300 2021/04/08 10:40:33 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.31.214 2021/04/08 10:40:33 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan" 2021/04/08 10:40:38 [WARN] raft: Heartbeat timeout from "" reached, starting election 2021/04/08 10:40:38 [INFO] raft: Node at 10.242.29.201:8300 [Candidate] entering Candidate state in term 2 2021/04/08 10:40:38 [INFO] raft: Election won. Tally: 2 2021/04/08 10:40:38 [INFO] raft: Node at 10.242.29.201:8300 [Leader] entering Leader state 2021/04/08 10:40:38 [INFO] raft: Added peer 98a759ec-9161-bdcc-82a8-d1b8e14f4785, starting replication 2021/04/08 10:40:38 [INFO] raft: Added peer 9eb7cc52-8937-e721-392d-855652f292f6, starting replication 2021/04/08 10:40:38 [INFO] consul: cluster leadership acquired 2021/04/08 10:40:38 [INFO] consul: New leader elected: dev-consul-server-0 2021/04/08 10:40:38 [INFO] raft: pipelining replication to peer {Voter 98a759ec-9161-bdcc-82a8-d1b8e14f4785 10.242.31.214:8300} 2021/04/08 10:40:38 [WARN] raft: AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300} rejected, sending older logs (next: 1) 2021/04/08 10:40:38 [INFO] raft: pipelining replication to peer {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300} 2021/04/08 10:40:38 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive 2021/04/08 10:40:41 [INFO] consul: member 'dev-consul-7467bf7b46-krcb7' joined, marking health alive 2021/04/08 10:40:41 [INFO] agent: Synced node info 2021/04/08 10:40:41 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/04/08 10:40:41 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive 2021/04/08 10:42:03 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 10:42:03 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 10:42:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 11:14:37 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 11:14:37 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 11:15:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.29.201:49846->10.242.53.14:8301: i/o timeout 2021/04/08 11:15:09 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/04/08 11:15:09 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:15:10 [WARN] raft: Failed to contact 9eb7cc52-8937-e721-392d-855652f292f6 in 2.500261994s 2021/04/08 11:15:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:15:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.29.201:49902->10.242.53.14:8301: i/o timeout 2021/04/08 11:15:12 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/04/08 11:15:12 [WARN] raft: Failed to contact 9eb7cc52-8937-e721-392d-855652f292f6 in 4.980255606s 2021/04/08 11:15:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:15:13 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/04/08 11:15:13 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.53.14 2021/04/08 11:15:13 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.53.14:8300) (DC: dc1) 2021/04/08 11:15:13 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical 2021/04/08 11:15:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:15:15 [WARN] raft: Failed to contact 9eb7cc52-8937-e721-392d-855652f292f6 in 7.478563961s 2021/04/08 11:15:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.29.201:48472->10.242.53.14:8302: i/o timeout 2021/04/08 11:15:15 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/04/08 11:15:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.29.201:49920->10.242.53.14:8301: i/o timeout 2021/04/08 11:15:15 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/04/08 11:15:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:15:18 [INFO] raft: aborting pipeline replication to peer {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300} 2021/04/08 11:15:18 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:15:18 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:42406->10.242.53.14:8300: i/o timeout 2021/04/08 11:15:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:15:19 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:15:20 [WARN] raft: Failed to contact 9eb7cc52-8937-e721-392d-855652f292f6 in 2.500302594s 2021/04/08 11:15:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:15:22 [WARN] raft: Failed to contact 9eb7cc52-8937-e721-392d-855652f292f6 in 4.946896613s 2021/04/08 11:15:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:15:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:15:25 [WARN] raft: Failed to contact 9eb7cc52-8937-e721-392d-855652f292f6 in 7.390689589s 2021/04/08 11:15:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.29.201:48624->10.242.53.14:8302: i/o timeout 2021/04/08 11:15:25 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/04/08 11:15:25 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.53.14:8301 2021/04/08 11:15:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:15:28 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:38716->10.242.53.14:8300: i/o timeout 2021/04/08 11:15:28 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:15:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:15:29 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:38740->10.242.53.14:8300: i/o timeout 2021/04/08 11:15:30 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:15:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:15:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:15:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:15:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.29.201:48708->10.242.53.14:8302: i/o timeout 2021/04/08 11:15:35 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/04/08 11:15:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:15:38 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:38860->10.242.53.14:8300: i/o timeout 2021/04/08 11:15:38 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:15:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:15:40 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:38914->10.242.53.14:8300: i/o timeout 2021/04/08 11:15:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:15:41 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:15:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:15:44 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 11:15:44 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.53.14:56928 2021/04/08 11:15:44 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 11:15:44 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/04/08 11:15:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.29.201:48968->10.242.53.14:8302: i/o timeout 2021/04/08 11:15:45 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/04/08 11:15:45 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/04/08 11:15:45 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.53.14 2021/04/08 11:15:45 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan" 2021/04/08 11:15:45 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:15:45 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 11:15:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:15:48 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:39048->10.242.53.14:8300: i/o timeout 2021/04/08 11:15:48 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:15:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:15:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:15:51 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:39094->10.242.53.14:8300: i/o timeout 2021/04/08 11:15:51 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:15:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:15:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:15:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:15:58 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:39222->10.242.53.14:8300: i/o timeout 2021/04/08 11:15:58 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:15:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:01 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:39286->10.242.53.14:8300: i/o timeout 2021/04/08 11:16:02 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:16:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:05 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.53.14:8301 2021/04/08 11:16:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:08 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:39402->10.242.53.14:8300: i/o timeout 2021/04/08 11:16:08 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:16:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:12 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:39478->10.242.53.14:8300: i/o timeout 2021/04/08 11:16:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:13 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:16:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:18 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:39572->10.242.53.14:8300: i/o timeout 2021/04/08 11:16:18 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:16:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:23 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:39652->10.242.53.14:8300: i/o timeout 2021/04/08 11:16:24 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:16:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:24 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 11:16:24 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/04/08 11:16:27 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:16:27 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 11:16:28 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:39730->10.242.53.14:8300: i/o timeout 2021/04/08 11:16:28 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:16:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:34 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:39822->10.242.53.14:8300: i/o timeout 2021/04/08 11:16:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:35 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:16:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:38 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:39902->10.242.53.14:8300: i/o timeout 2021/04/08 11:16:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:39 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:16:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:45 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:40014->10.242.53.14:8300: i/o timeout 2021/04/08 11:16:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:47 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:16:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:49 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:40072->10.242.53.14:8300: i/o timeout 2021/04/08 11:16:50 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:16:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:55 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.53.14:8302 2021/04/08 11:16:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:16:57 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:40188->10.242.53.14:8300: i/o timeout 2021/04/08 11:16:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:17:00 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:17:00 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:40246->10.242.53.14:8300: i/o timeout 2021/04/08 11:17:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:17:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:17:03 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:17:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:17:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:17:06 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 11:17:06 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/04/08 11:17:09 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:17:09 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 11:17:10 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:40416->10.242.53.14:8300: i/o timeout 2021/04/08 11:17:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:17:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:17:13 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:40452->10.242.53.14:8300: i/o timeout 2021/04/08 11:17:13 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.53.14 2021/04/08 11:17:13 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan" 2021/04/08 11:17:13 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.53.14 2021/04/08 11:17:13 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.53.14:8300) (DC: dc1) 2021/04/08 11:17:13 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/04/08 11:17:13 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/04/08 11:17:14 [DEBUG] raft-net: 10.242.29.201:8300 accepted connection from: 10.242.53.14:55524 2021/04/08 11:17:14 [WARN] raft: Rejecting vote request from 10.242.53.14:8300 since we have a leader: 10.242.29.201:8300 2021/04/08 11:17:18 [INFO] raft: pipelining replication to peer {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300} 2021/04/08 11:18:19 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:18:19 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 11:18:21 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:18:21 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 11:18:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.29.201:52996->10.242.53.14:8301: i/o timeout 2021/04/08 11:18:22 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/04/08 11:18:22 [WARN] raft: Failed to contact 9eb7cc52-8937-e721-392d-855652f292f6 in 2.500224483s 2021/04/08 11:18:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:18:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:18:25 [WARN] raft: Failed to contact 9eb7cc52-8937-e721-392d-855652f292f6 in 4.994556407s 2021/04/08 11:18:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.29.201:53068->10.242.53.14:8301: i/o timeout 2021/04/08 11:18:26 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/04/08 11:18:26 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/04/08 11:18:26 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.53.14 2021/04/08 11:18:26 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.53.14:8300) (DC: dc1) 2021/04/08 11:18:26 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical 2021/04/08 11:18:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:18:27 [WARN] raft: Failed to contact 9eb7cc52-8937-e721-392d-855652f292f6 in 7.447446688s 2021/04/08 11:18:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:18:30 [INFO] raft: aborting pipeline replication to peer {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300} 2021/04/08 11:18:30 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:18:30 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:40710->10.242.53.14:8300: i/o timeout 2021/04/08 11:18:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:18:31 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:18:32 [WARN] raft: Failed to contact 9eb7cc52-8937-e721-392d-855652f292f6 in 2.500201411s 2021/04/08 11:18:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:18:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:18:35 [WARN] raft: Failed to contact 9eb7cc52-8937-e721-392d-855652f292f6 in 4.978235924s 2021/04/08 11:18:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.29.201:51726->10.242.53.14:8302: i/o timeout 2021/04/08 11:18:35 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/04/08 11:18:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:18:37 [WARN] raft: Failed to contact 9eb7cc52-8937-e721-392d-855652f292f6 in 7.471250892s 2021/04/08 11:18:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:18:40 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:41848->10.242.53.14:8300: i/o timeout 2021/04/08 11:18:40 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:18:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:18:41 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:41860->10.242.53.14:8300: i/o timeout 2021/04/08 11:18:42 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:18:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:18:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:18:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.29.201:51888->10.242.53.14:8302: i/o timeout 2021/04/08 11:18:45 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/04/08 11:18:45 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.53.14:8301 2021/04/08 11:18:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:18:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:18:49 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 11:18:49 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/04/08 11:18:50 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:42016->10.242.53.14:8300: i/o timeout 2021/04/08 11:18:50 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:18:51 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:18:52 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:42038->10.242.53.14:8300: i/o timeout 2021/04/08 11:18:52 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:18:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:18:53 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 11:18:53 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.53.14:55510 2021/04/08 11:18:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:18:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.29.201:52068->10.242.53.14:8302: i/o timeout 2021/04/08 11:18:55 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/04/08 11:18:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:18:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:00 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:42176->10.242.53.14:8300: i/o timeout 2021/04/08 11:19:00 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:19:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:02 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:42218->10.242.53.14:8300: i/o timeout 2021/04/08 11:19:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:03 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:19:04 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.53.14 2021/04/08 11:19:04 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan" 2021/04/08 11:19:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.29.201:52226->10.242.53.14:8302: i/o timeout 2021/04/08 11:19:05 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/04/08 11:19:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:10 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:42354->10.242.53.14:8300: i/o timeout 2021/04/08 11:19:10 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:19:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:13 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:42410->10.242.53.14:8300: i/o timeout 2021/04/08 11:19:14 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:19:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:20 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:42504->10.242.53.14:8300: i/o timeout 2021/04/08 11:19:20 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:19:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:24 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:42566->10.242.53.14:8300: i/o timeout 2021/04/08 11:19:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:25 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:19:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:30 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:42654->10.242.53.14:8300: i/o timeout 2021/04/08 11:19:30 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:19:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:30 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 11:19:30 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/04/08 11:19:33 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:19:33 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 11:19:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:35 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:42726->10.242.53.14:8300: i/o timeout 2021/04/08 11:19:36 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:19:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:40 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:42812->10.242.53.14:8300: i/o timeout 2021/04/08 11:19:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:41 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:19:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:46 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:42894->10.242.53.14:8300: i/o timeout 2021/04/08 11:19:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:47 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:19:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:51 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:42964->10.242.53.14:8300: i/o timeout 2021/04/08 11:19:51 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:19:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:57 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:43068->10.242.53.14:8300: i/o timeout 2021/04/08 11:19:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:19:59 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:20:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:20:01 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:43124->10.242.53.14:8300: i/o timeout 2021/04/08 11:20:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:20:03 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:20:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:20:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:20:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:20:09 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:43266->10.242.53.14:8300: i/o timeout 2021/04/08 11:20:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:20:11 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.53.14 2021/04/08 11:20:11 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.53.14:8300) (DC: dc1) 2021/04/08 11:20:11 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/04/08 11:20:11 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.53.14 2021/04/08 11:20:11 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan" 2021/04/08 11:20:11 [INFO] raft: pipelining replication to peer {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300} 2021/04/08 11:20:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.29.201:55494->10.242.53.14:8301: i/o timeout 2021/04/08 11:20:59 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:20:59 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 11:21:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.29.201:55516->10.242.53.14:8301: i/o timeout 2021/04/08 11:21:01 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/04/08 11:21:01 [WARN] raft: Failed to contact 9eb7cc52-8937-e721-392d-855652f292f6 in 2.500254282s 2021/04/08 11:21:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.29.201:55576->10.242.53.14:8301: i/o timeout 2021/04/08 11:21:04 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/04/08 11:21:04 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.53.14 2021/04/08 11:21:04 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.53.14:8300) (DC: dc1) 2021/04/08 11:21:04 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical 2021/04/08 11:21:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:05 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:21:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:07 [WARN] raft: Failed to contact 9eb7cc52-8937-e721-392d-855652f292f6 in 2.500198772s 2021/04/08 11:21:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:10 [WARN] raft: Failed to contact 9eb7cc52-8937-e721-392d-855652f292f6 in 4.973734578s 2021/04/08 11:21:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:12 [WARN] raft: Failed to contact 9eb7cc52-8937-e721-392d-855652f292f6 in 7.423546223s 2021/04/08 11:21:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:15 [INFO] raft: aborting pipeline replication to peer {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300} 2021/04/08 11:21:15 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:21:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.29.201:54250->10.242.53.14:8302: i/o timeout 2021/04/08 11:21:15 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/04/08 11:21:15 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:43482->10.242.53.14:8300: i/o timeout 2021/04/08 11:21:16 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:21:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:17 [WARN] raft: Failed to contact 9eb7cc52-8937-e721-392d-855652f292f6 in 2.500217104s 2021/04/08 11:21:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:20 [WARN] raft: Failed to contact 9eb7cc52-8937-e721-392d-855652f292f6 in 4.960129949s 2021/04/08 11:21:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:21 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 11:21:21 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/04/08 11:21:21 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 11:21:21 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.53.14:57020 2021/04/08 11:21:22 [WARN] raft: Failed to contact 9eb7cc52-8937-e721-392d-855652f292f6 in 7.455229897s 2021/04/08 11:21:23 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:21:23 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 11:21:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:25 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:44454->10.242.53.14:8300: i/o timeout 2021/04/08 11:21:25 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:21:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.29.201:54398->10.242.53.14:8302: i/o timeout 2021/04/08 11:21:25 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/04/08 11:21:25 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.53.14:8301 2021/04/08 11:21:26 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:44468->10.242.53.14:8300: i/o timeout 2021/04/08 11:21:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:27 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:21:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:35 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:44604->10.242.53.14:8300: i/o timeout 2021/04/08 11:21:35 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:21:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:37 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:44634->10.242.53.14:8300: i/o timeout 2021/04/08 11:21:38 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:21:38 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/04/08 11:21:38 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.53.14 2021/04/08 11:21:38 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan" 2021/04/08 11:21:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.29.201:54660->10.242.53.14:8302: i/o timeout 2021/04/08 11:21:40 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/04/08 11:21:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:43 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.53.14:8302 2021/04/08 11:21:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:45 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:44780->10.242.53.14:8300: i/o timeout 2021/04/08 11:21:45 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:21:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:48 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:44820->10.242.53.14:8300: i/o timeout 2021/04/08 11:21:48 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:21:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:55 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:44940->10.242.53.14:8300: i/o timeout 2021/04/08 11:21:55 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:21:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:58 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:45040->10.242.53.14:8300: i/o timeout 2021/04/08 11:21:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:21:59 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:22:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:02 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 11:22:02 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/04/08 11:22:05 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:45166->10.242.53.14:8300: i/o timeout 2021/04/08 11:22:05 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:22:05 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.53.14:8301 2021/04/08 11:22:05 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:22:05 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 11:22:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:09 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:45242->10.242.53.14:8300: i/o timeout 2021/04/08 11:22:10 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:22:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:15 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:45350->10.242.53.14:8300: i/o timeout 2021/04/08 11:22:15 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:22:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:20 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:45432->10.242.53.14:8300: i/o timeout 2021/04/08 11:22:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:21 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:22:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:25 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:45560->10.242.53.14:8300: i/o timeout 2021/04/08 11:22:25 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:22:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:31 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:45642->10.242.53.14:8300: i/o timeout 2021/04/08 11:22:32 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:22:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:35 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:45732->10.242.53.14:8300: i/o timeout 2021/04/08 11:22:36 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:22:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:42 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:45870->10.242.53.14:8300: i/o timeout 2021/04/08 11:22:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:43 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.53.14:8302 2021/04/08 11:22:44 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:22:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:44 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 11:22:44 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/04/08 11:22:46 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:46020->10.242.53.14:8300: i/o timeout 2021/04/08 11:22:47 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:22:47 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:22:47 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 11:22:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:54 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:46336->10.242.53.14:8300: i/o timeout 2021/04/08 11:22:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:22:57 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:46400->10.242.53.14:8300: i/o timeout 2021/04/08 11:22:58 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:22:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:00 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:23:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:08 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:46600->10.242.53.14:8300: i/o timeout 2021/04/08 11:23:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:10 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:46658->10.242.53.14:8300: i/o timeout 2021/04/08 11:23:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:14 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:23:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:15 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:23:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:24 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:46932->10.242.53.14:8300: i/o timeout 2021/04/08 11:23:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:25 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:46976->10.242.53.14:8300: i/o timeout 2021/04/08 11:23:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:26 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 11:23:26 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/04/08 11:23:29 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:23:29 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 11:23:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:35 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:23:35 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:23:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:43 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.53.14:8302 2021/04/08 11:23:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:45 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:47370->10.242.53.14:8300: i/o timeout 2021/04/08 11:23:45 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:47378->10.242.53.14:8300: i/o timeout 2021/04/08 11:23:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:56 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:23:56 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:23:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:23:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:06 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:47786->10.242.53.14:8300: i/o timeout 2021/04/08 11:24:06 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:47790->10.242.53.14:8300: i/o timeout 2021/04/08 11:24:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:08 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 11:24:08 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/04/08 11:24:11 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:24:11 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 11:24:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:16 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:24:16 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:24:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:26 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:48266->10.242.53.14:8300: i/o timeout 2021/04/08 11:24:26 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:48282->10.242.53.14:8300: i/o timeout 2021/04/08 11:24:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:36 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:24:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:37 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:24:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:43 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.53.14:8302 2021/04/08 11:24:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:46 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:48664->10.242.53.14:8300: i/o timeout 2021/04/08 11:24:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:47 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:48692->10.242.53.14:8300: i/o timeout 2021/04/08 11:24:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:50 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 11:24:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:51 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/04/08 11:24:53 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:24:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:56 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:24:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:24:59 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:25:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:06 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:49184->10.242.53.14:8300: i/o timeout 2021/04/08 11:25:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:09 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:49276->10.242.53.14:8300: i/o timeout 2021/04/08 11:25:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:16 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:25:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:20 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:25:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:26 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:49852->10.242.53.14:8300: i/o timeout 2021/04/08 11:25:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:30 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:50030->10.242.53.14:8300: i/o timeout 2021/04/08 11:25:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:32 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 11:25:32 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/04/08 11:25:35 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:25:35 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 11:25:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:37 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:25:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:41 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:25:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:43 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.53.14:8302 2021/04/08 11:25:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:45 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.53.14:8301 2021/04/08 11:25:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:47 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:50440->10.242.53.14:8300: i/o timeout 2021/04/08 11:25:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:51 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:50580->10.242.53.14:8300: i/o timeout 2021/04/08 11:25:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:25:57 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:25:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:02 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:26:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:07 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:50890->10.242.53.14:8300: i/o timeout 2021/04/08 11:26:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:12 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:50970->10.242.53.14:8300: i/o timeout 2021/04/08 11:26:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:14 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 11:26:14 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/04/08 11:26:17 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:26:17 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:26:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:23 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:26:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:27 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:51260->10.242.53.14:8300: i/o timeout 2021/04/08 11:26:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:33 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: read tcp 10.242.29.201:51334->10.242.53.14:8300: i/o timeout 2021/04/08 11:26:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:37 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:26:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:44 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:26:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:47 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: read tcp 10.242.29.201:51612->10.242.53.14:8300: i/o timeout 2021/04/08 11:26:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:26:53 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.53.14 2021/04/08 11:26:53 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.53.14:8300) (DC: dc1) 2021/04/08 11:26:53 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/04/08 11:26:53 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.53.14 2021/04/08 11:26:53 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan" 2021/04/08 11:26:53 [WARN] raft: Rejecting vote request from 10.242.53.14:8300 since we have a leader: 10.242.29.201:8300 2021/04/08 11:26:58 [INFO] raft: pipelining replication to peer {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300} 2021/04/08 11:27:01 [INFO] raft: aborting pipeline replication to peer {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300} 2021/04/08 11:27:01 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: EOF 2021/04/08 11:27:01 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: dial tcp ->10.242.53.14:8300: getsockopt: connection refused 2021/04/08 11:27:01 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: dial tcp ->10.242.53.14:8300: getsockopt: connection refused 2021/04/08 11:27:01 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: dial tcp ->10.242.53.14:8300: getsockopt: connection refused 2021/04/08 11:27:02 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: dial tcp ->10.242.53.14:8300: getsockopt: connection refused 2021/04/08 11:27:02 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: dial tcp ->10.242.53.14:8300: getsockopt: connection refused 2021/04/08 11:27:02 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: dial tcp ->10.242.53.14:8300: getsockopt: connection refused 2021/04/08 11:27:02 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: dial tcp ->10.242.53.14:8300: getsockopt: connection refused 2021/04/08 11:27:02 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: dial tcp ->10.242.53.14:8300: getsockopt: connection refused 2021/04/08 11:27:02 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/04/08 11:27:02 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: dial tcp ->10.242.53.14:8300: getsockopt: connection refused 2021/04/08 11:27:03 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: stream closed 2021/04/08 11:27:03 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: dial tcp ->10.242.53.14:8300: getsockopt: connection refused 2021/04/08 11:27:03 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: dial tcp ->10.242.53.14:8300: getsockopt: connection refused 2021/04/08 11:27:03 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: dial tcp ->10.242.53.14:8300: getsockopt: connection refused 2021/04/08 11:27:04 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:27:04 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 11:27:04 [WARN] raft: Failed to contact 9eb7cc52-8937-e721-392d-855652f292f6 in 2.500266861s 2021/04/08 11:27:04 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: dial tcp ->10.242.53.14:8300: getsockopt: connection refused 2021/04/08 11:27:04 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: dial tcp ->10.242.53.14:8300: getsockopt: connection refused 2021/04/08 11:27:05 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error getting client: failed to get conn: dial tcp ->10.242.53.14:8300: getsockopt: connection refused 2021/04/08 11:27:05 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:27:05 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 11:27:06 [WARN] raft: Failed to contact 9eb7cc52-8937-e721-392d-855652f292f6 in 4.992812983s 2021/04/08 11:27:06 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/04/08 11:27:06 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.53.14 2021/04/08 11:27:06 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.53.14:8300) (DC: dc1) 2021/04/08 11:27:06 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical 2021/04/08 11:27:07 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:27:07 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:27:07 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 11:27:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:27:09 [WARN] raft: Failed to contact 9eb7cc52-8937-e721-392d-855652f292f6 in 7.492705196s 2021/04/08 11:27:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:27:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:27:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:27:15 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:27:16 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:27:16 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/04/08 11:27:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:27:16 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error getting client: failed to get conn: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:27:17 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:27:19 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:27:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:27:22 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:27:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:27:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:27:26 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:27:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:27:27 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:27:28 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error getting client: failed to get conn: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:27:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:27:31 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:27:32 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:27:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:27:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:27:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:27:37 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:27:38 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:27:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:27:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:27:40 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error getting client: failed to get conn: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:27:42 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:27:43 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:27:43 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 11:27:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:27:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:27:48 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:27:48 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/04/08 11:27:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:27:50 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:27:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:27:52 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:27:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:27:52 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error getting client: failed to get conn: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:27:55 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:27:55 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 11:27:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:27:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:28:00 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:28:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:28:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:28:03 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:28:04 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:28:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:28:04 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error getting client: failed to get conn: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:28:07 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.53.14 2021/04/08 11:28:07 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan" 2021/04/08 11:28:07 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:28:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:28:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:28:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:28:13 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:28:13 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.53.14:8302 2021/04/08 11:28:14 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:28:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:28:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:28:16 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error getting client: failed to get conn: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:28:19 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 11:28:20 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:28:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:28:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/04/08 11:28:23 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:28:24 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.251.158 2021/04/08 11:28:24 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.251.158:8300) (DC: dc1) 2021/04/08 11:28:24 [INFO] raft: Updating configuration with AddNonvoter (a4860686-c5d1-721e-3947-0568f46d90b5, 10.242.251.158:8300) to [{Suffrage:Voter ID:98a759ec-9161-bdcc-82a8-d1b8e14f4785 Address:10.242.31.214:8300} {Suffrage:Voter ID:8c9b3e16-04a3-1dc7-b2b3-e1ac04ac3526 Address:10.242.29.201:8300} {Suffrage:Voter ID:9eb7cc52-8937-e721-392d-855652f292f6 Address:10.242.53.14:8300} {Suffrage:Nonvoter ID:a4860686-c5d1-721e-3947-0568f46d90b5 Address:10.242.251.158:8300}] 2021/04/08 11:28:24 [INFO] raft: Added peer a4860686-c5d1-721e-3947-0568f46d90b5, starting replication 2021/04/08 11:28:24 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/04/08 11:28:24 [ERR] memberlist: Conflicting address for dev-consul-server-1.dc1. Mine: 10.242.53.14:8302 Theirs: 10.242.251.158:8302 2021/04/08 11:28:24 [WARN] serf: Name conflict for 'dev-consul-server-1.dc1' both 10.242.53.14:8302 and 10.242.251.158:8302 are claiming 2021/04/08 11:28:24 [WARN] raft: AppendEntries to {Nonvoter a4860686-c5d1-721e-3947-0568f46d90b5 10.242.251.158:8300} rejected, sending older logs (next: 1) 2021/04/08 11:28:25 [INFO] raft: pipelining replication to peer {Nonvoter a4860686-c5d1-721e-3947-0568f46d90b5 10.242.251.158:8300} 2021/04/08 11:28:25 [ERR] memberlist: Conflicting address for dev-consul-server-1.dc1. Mine: 10.242.53.14:8302 Theirs: 10.242.251.158:8302 2021/04/08 11:28:25 [WARN] serf: Name conflict for 'dev-consul-server-1.dc1' both 10.242.53.14:8302 and 10.242.251.158:8302 are claiming 2021/04/08 11:28:25 [ERR] memberlist: Conflicting address for dev-consul-server-1.dc1. Mine: 10.242.53.14:8302 Theirs: 10.242.251.158:8302 2021/04/08 11:28:25 [WARN] serf: Name conflict for 'dev-consul-server-1.dc1' both 10.242.53.14:8302 and 10.242.251.158:8302 are claiming 2021/04/08 11:28:25 [ERR] memberlist: Conflicting address for dev-consul-server-1.dc1. Mine: 10.242.53.14:8302 Theirs: 10.242.251.158:8302 2021/04/08 11:28:25 [WARN] serf: Name conflict for 'dev-consul-server-1.dc1' both 10.242.53.14:8302 and 10.242.251.158:8302 are claiming 2021/04/08 11:28:25 [ERR] memberlist: Conflicting address for dev-consul-server-1.dc1. Mine: 10.242.53.14:8302 Theirs: 10.242.251.158:8302 2021/04/08 11:28:25 [WARN] serf: Name conflict for 'dev-consul-server-1.dc1' both 10.242.53.14:8302 and 10.242.251.158:8302 are claiming 2021/04/08 11:28:28 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/04/08 11:28:28 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error getting client: failed to get conn: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:28:30 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:28:33 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:28:38 [INFO] autopilot: Promoting Server (ID: "a4860686-c5d1-721e-3947-0568f46d90b5" Address: "10.242.251.158:8300") to voter 2021/04/08 11:28:38 [INFO] raft: Updating configuration with AddStaging (a4860686-c5d1-721e-3947-0568f46d90b5, 10.242.251.158:8300) to [{Suffrage:Voter ID:98a759ec-9161-bdcc-82a8-d1b8e14f4785 Address:10.242.31.214:8300} {Suffrage:Voter ID:8c9b3e16-04a3-1dc7-b2b3-e1ac04ac3526 Address:10.242.29.201:8300} {Suffrage:Voter ID:9eb7cc52-8937-e721-392d-855652f292f6 Address:10.242.53.14:8300} {Suffrage:Voter ID:a4860686-c5d1-721e-3947-0568f46d90b5 Address:10.242.251.158:8300}] 2021/04/08 11:28:39 [INFO] autopilot: Attempting removal of stale Server (ID: "9eb7cc52-8937-e721-392d-855652f292f6" Address: "10.242.53.14:8300") 2021/04/08 11:28:39 [INFO] raft: Updating configuration with RemoveServer (9eb7cc52-8937-e721-392d-855652f292f6, ) to [{Suffrage:Voter ID:98a759ec-9161-bdcc-82a8-d1b8e14f4785 Address:10.242.31.214:8300} {Suffrage:Voter ID:8c9b3e16-04a3-1dc7-b2b3-e1ac04ac3526 Address:10.242.29.201:8300} {Suffrage:Voter ID:a4860686-c5d1-721e-3947-0568f46d90b5 Address:10.242.251.158:8300}] 2021/04/08 11:28:39 [INFO] raft: Removed peer 9eb7cc52-8937-e721-392d-855652f292f6, stopping replication after 596 2021/04/08 11:28:41 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:28:43 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:28:51 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:28:53 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:29:02 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:29:03 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:29:12 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:29:13 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:29:13 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.53.14:8302 2021/04/08 11:29:23 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:29:24 [WARN] Unable to get address for server id 9eb7cc52-8937-e721-392d-855652f292f6, using fallback address 10.242.53.14:8300: Could not find address for server id 9eb7cc52-8937-e721-392d-855652f292f6 2021/04/08 11:29:33 [ERR] raft: Failed to heartbeat to 10.242.53.14:8300: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:29:34 [ERR] raft: Failed to AppendEntries to {Voter 9eb7cc52-8937-e721-392d-855652f292f6 10.242.53.14:8300}: dial tcp ->10.242.53.14:8300: i/o timeout 2021/04/08 11:30:20 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.251.158 2021/04/08 11:30:20 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"