By type
+ CONSUL_BIND= + [ -n ] + CONSUL_CLIENT= + [ -n ] + CONSUL_DATA_DIR=/consul/data + CONSUL_CONFIG_DIR=/consul/config + [ -n ] + [ a = - ] + [ agent = agent ] + shift + set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui + [ consul = consul ] + stat -c %u /consul/data + id -u consul + [ 100 != 100 ] + stat -c %u /consul/config + id -u consul + [ 100 != 100 ] + [ ! -z ] + exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui bootstrap_expect > 0: expecting 3 servers ==> Starting Consul agent... ==> Consul agent running! Version: 'v1.0.6' Node ID: 'ebda65a7-b47f-c28e-e8cd-05aadf21320c' Node name: 'dev-consul-server-2' Datacenter: 'dc1' (Segment: '
') Server: true (Bootstrap: false) Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600) Cluster Addr: 10.242.224.74 (LAN: 8301, WAN: 8302) Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false ==> Log data will now stream in as it occurs: 2021/06/17 10:45:25 [INFO] raft: Initial configuration (index=0): [] 2021/06/17 10:45:25 [INFO] raft: Node at 10.242.224.74:8300 [Follower] entering Follower state (Leader: "") 2021/06/17 10:45:25 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.224.74 2021/06/17 10:45:25 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.224.74 2021/06/17 10:45:25 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan" 2021/06/17 10:45:25 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.224.74:8300) (DC: dc1) 2021/06/17 10:45:25 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp) 2021/06/17 10:45:25 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp) 2021/06/17 10:45:25 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server] 2021/06/17 10:45:25 [INFO] agent: Started HTTP server on [::]:8500 (tcp) 2021/06/17 10:45:25 [INFO] agent: started state syncer 2021/06/17 10:45:25 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer 2021/06/17 10:45:25 [INFO] agent: Joining LAN cluster... 2021/06/17 10:45:25 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.71.138 2021/06/17 10:45:25 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.105.8 2021/06/17 10:45:25 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.71.138:8300) (DC: dc1) 2021/06/17 10:45:25 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.242.224.74:8300,10.242.71.138:8300,10.242.105.8:8300 2021/06/17 10:45:25 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.105.8:8300) (DC: dc1) 2021/06/17 10:45:25 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.105.8 2021/06/17 10:45:25 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/17 10:45:25 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.71.138 2021/06/17 10:45:25 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan" 2021/06/17 10:45:25 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host 2021/06/17 10:45:25 [INFO] agent: (LAN) joined: 2 Err: 2021/06/17 10:45:25 [INFO] agent: Join LAN completed. Synced with 2 initial agents 2021/06/17 10:45:32 [ERR] agent: failed to sync remote state: No cluster leader 2021/06/17 10:45:33 [WARN] raft: Heartbeat timeout from "" reached, starting election 2021/06/17 10:45:33 [INFO] raft: Node at 10.242.224.74:8300 [Candidate] entering Candidate state in term 2 2021/06/17 10:45:33 [INFO] raft: Election won. Tally: 2 2021/06/17 10:45:33 [INFO] raft: Node at 10.242.224.74:8300 [Leader] entering Leader state 2021/06/17 10:45:33 [INFO] raft: Added peer a75e2fc6-3d9c-d681-67ad-f6786fe9e276, starting replication 2021/06/17 10:45:33 [INFO] raft: Added peer 080c2ccf-91d8-bfc6-205b-266925ed7eaa, starting replication 2021/06/17 10:45:33 [INFO] consul: cluster leadership acquired 2021/06/17 10:45:33 [INFO] consul: New leader elected: dev-consul-server-2 2021/06/17 10:45:33 [WARN] raft: AppendEntries to {Voter 080c2ccf-91d8-bfc6-205b-266925ed7eaa 10.242.105.8:8300} rejected, sending older logs (next: 1) 2021/06/17 10:45:33 [INFO] raft: pipelining replication to peer {Voter 080c2ccf-91d8-bfc6-205b-266925ed7eaa 10.242.105.8:8300} 2021/06/17 10:45:33 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive 2021/06/17 10:45:33 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive 2021/06/17 10:45:33 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/06/17 10:45:35 [WARN] raft: Failed to contact a75e2fc6-3d9c-d681-67ad-f6786fe9e276 in 2.501808032s 2021/06/17 10:45:35 [INFO] agent: Synced node info 2021/06/17 10:45:36 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:45:36 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:45:37 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:45:38 [WARN] raft: Failed to contact a75e2fc6-3d9c-d681-67ad-f6786fe9e276 in 4.997363558s 2021/06/17 10:45:39 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:45:41 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:45:43 [ERR] raft: Failed to make RequestVote RPC to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:42376->10.242.71.138:8300: i/o timeout 2021/06/17 10:45:43 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:42380->10.242.71.138:8300: i/o timeout 2021/06/17 10:45:43 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:45:43 [WARN] raft: AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300} rejected, sending older logs (next: 1) 2021/06/17 10:45:43 [INFO] raft: pipelining replication to peer {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300} ==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=925696a3-fe11-5d1f-d495-4c74879675ab&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 2021/06/17 10:45:58 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:45:58 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:45:59 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:46:01 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:46:08 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:46:08 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:46:09 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:46:18 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:46:18 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:46:19 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:46:21 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:46:23 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:46:25 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:46:25 [INFO] raft: aborting pipeline replication to peer {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300} 2021/06/17 10:46:27 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:46:28 [INFO] raft: pipelining replication to peer {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300} 2021/06/17 10:46:30 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:46:30 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:46:31 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:46:33 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:46:36 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:46:37 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:46:40 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:46:41 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:46:43 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:46:45 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:46:47 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:46:47 [ERR] raft: Failed to pipeline AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: write tcp 10.242.224.74:42452->10.242.71.138:8300: use of closed network connection 2021/06/17 10:46:47 [INFO] raft: aborting pipeline replication to peer {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300} 2021/06/17 10:46:50 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:46:51 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:46:53 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:46:55 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:46:57 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:46:57 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:42710->10.242.71.138:8300: i/o timeout 2021/06/17 10:46:59 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:47:01 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:47:03 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:47:05 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:47:07 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:42860->10.242.71.138:8300: i/o timeout 2021/06/17 10:47:08 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:47:08 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:47:09 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:47:11 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:47:14 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:47:15 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:47:17 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:47:17 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:42934->10.242.71.138:8300: i/o timeout 2021/06/17 10:47:19 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:47:21 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:47:22 [INFO] raft: pipelining replication to peer {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300} 2021/06/17 10:47:24 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:47:25 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:47:32 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:47:32 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:47:34 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:47:34 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:47:35 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:47:37 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:47:39 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:47:41 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:47:48 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:47:49 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-tm7mq 10.242.71.136 2021/06/17 10:47:49 [INFO] consul: member 'dev-consul-7467bf7b46-tm7mq' joined, marking health alive 2021/06/17 10:47:52 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:47:52 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:47:54 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:47:54 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/06/17 10:47:55 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:47:57 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:47:59 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:48:02 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:48:03 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:48:32 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:48:32 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:48:33 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:48:35 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:48:38 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:48:38 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:48:42 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:48:43 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:48:45 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:48:47 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:48:50 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:48:51 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:48:53 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:49:02 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:49:03 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:49:08 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:49:09 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:49:22 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:49:22 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:49:23 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:49:25 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:49:27 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:49:30 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:49:30 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:49:31 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:49:34 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:49:34 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:49:35 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:49:37 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:49:42 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:49:43 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:50:02 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:50:03 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:50:05 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:50:07 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:50:12 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:50:13 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:50:22 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:50:22 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:50:22 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/06/17 10:50:23 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:50:25 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:50:30 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:50:30 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:50:31 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:50:38 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:50:39 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:50:58 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:50:58 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:50:59 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:52:04 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:52:04 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:52:05 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:52:07 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:52:38 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:52:44 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:52:44 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:52:45 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:52:47 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:52:54 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:52:54 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:52:55 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:53:10 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:53:10 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:53:11 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:53:34 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:53:35 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:53:37 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:53:42 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:53:43 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:54:12 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:54:13 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:54:15 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:54:17 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:54:24 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:54:24 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/06/17 10:54:25 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:55:10 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:56:18 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:56:24 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:56:26 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:56:26 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:56:27 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:56:30 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:56:32 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:57:48 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:57:48 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:57:49 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:57:58 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:58:04 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:58:12 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:58:12 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:58:48 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:58:49 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 10:58:58 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:59:28 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:59:28 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 10:59:58 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 10:59:58 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:00:14 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:00:15 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:00:28 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:00:28 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:00:29 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:03:48 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:03:48 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:03:49 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:04:08 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:04:09 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:04:11 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:04:13 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:04:38 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:04:38 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:04:58 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:04:58 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:04:59 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:05:01 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:05:03 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:05:05 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:05:06 [INFO] raft: aborting pipeline replication to peer {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300} 2021/06/17 11:05:07 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:05:09 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:05:11 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:05:13 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:05:15 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:05:16 [INFO] raft: pipelining replication to peer {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300} 2021/06/17 11:06:04 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:06:04 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:06:05 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:06:08 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:06:08 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:06:09 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:06:14 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:06:15 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:06:17 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:06:20 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:06:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:06:21 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:06:23 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:07:14 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:07:22 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:07:23 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:07:52 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:07:52 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:07:53 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:07:56 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:07:57 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:07:59 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:08:14 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:08:14 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:08:15 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:08:56 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:09:00 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:09:26 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:09:36 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:10:04 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:10:04 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:10:05 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:10:07 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:10:14 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:10:14 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:10:20 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:10:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:11:06 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:11:07 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:11:09 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:11:36 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:11:36 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:11:37 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:12:26 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:12:26 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:12:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/06/17 11:12:27 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:12:40 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:13:06 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:13:06 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:13:40 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:13:40 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:13:40 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/06/17 11:15:10 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:15:50 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:41:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.224.74:33352->10.242.71.136:8301: i/o timeout 2021/06/17 11:41:11 [INFO] memberlist: Suspect dev-consul-7467bf7b46-tm7mq has failed, no acks received 2021/06/17 11:41:14 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:41:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.224.74:33384->10.242.71.136:8301: i/o timeout 2021/06/17 11:41:14 [INFO] memberlist: Suspect dev-consul-7467bf7b46-tm7mq has failed, no acks received 2021/06/17 11:41:15 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:41:15 [WARN] raft: Failed to contact a75e2fc6-3d9c-d681-67ad-f6786fe9e276 in 2.500207233s 2021/06/17 11:41:16 [INFO] memberlist: Marking dev-consul-7467bf7b46-tm7mq as failed, suspect timeout reached (2 peer confirmations) 2021/06/17 11:41:16 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-tm7mq 10.242.71.136 2021/06/17 11:41:16 [INFO] consul: member 'dev-consul-7467bf7b46-tm7mq' failed, marking health critical 2021/06/17 11:41:17 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:41:17 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured 2021/06/17 11:41:19 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:41:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.224.74:47616->10.242.71.138:8301: i/o timeout 2021/06/17 11:41:20 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/17 11:41:20 [WARN] raft: Failed to contact a75e2fc6-3d9c-d681-67ad-f6786fe9e276 in 2.500252498s 2021/06/17 11:41:21 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:41:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.224.74:47624->10.242.71.138:8301: i/o timeout 2021/06/17 11:41:22 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/17 11:41:23 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:41:23 [WARN] raft: Failed to contact a75e2fc6-3d9c-d681-67ad-f6786fe9e276 in 2.500183517s 2021/06/17 11:41:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.224.74:47656->10.242.71.138:8301: i/o timeout 2021/06/17 11:41:24 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/17 11:41:25 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:41:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.224.74:52390->10.242.71.138:8302: i/o timeout 2021/06/17 11:41:25 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/06/17 11:41:25 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1) 2021/06/17 11:41:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.224.74:47674->10.242.71.138:8301: i/o timeout 2021/06/17 11:41:26 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/17 11:41:26 [ERR] raft: Failed to pipeline AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: write tcp 10.242.224.74:43064->10.242.71.138:8300: use of closed network connection 2021/06/17 11:41:26 [INFO] raft: aborting pipeline replication to peer {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300} 2021/06/17 11:41:28 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:41:28 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:41:29 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:41:29 [ERR] memberlist: failed to receive: read tcp 10.242.224.74:8301->10.242.71.136:44422: i/o timeout from=10.242.71.136:44422 2021/06/17 11:41:29 [ERR] memberlist: Failed to send error: write tcp 10.242.224.74:8301->10.242.71.136:44422: i/o timeout from=10.242.71.136:44422 2021/06/17 11:41:29 [WARN] raft: Failed to contact a75e2fc6-3d9c-d681-67ad-f6786fe9e276 in 2.500306314s 2021/06/17 11:41:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.224.74:47700->10.242.71.138:8301: i/o timeout 2021/06/17 11:41:29 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/17 11:41:30 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations) 2021/06/17 11:41:30 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.71.138 2021/06/17 11:41:30 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.71.138:8300) (DC: dc1) 2021/06/17 11:41:30 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical 2021/06/17 11:41:31 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:41:31 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2) 2021/06/17 11:41:31 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.71.138 2021/06/17 11:41:31 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.71.138:8300) (DC: dc1) 2021/06/17 11:41:31 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive 2021/06/17 11:41:31 [WARN] raft: Failed to contact a75e2fc6-3d9c-d681-67ad-f6786fe9e276 in 4.931898135s 2021/06/17 11:41:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.224.74:47728->10.242.71.138:8301: i/o timeout 2021/06/17 11:41:32 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/17 11:41:33 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:41:34 [WARN] raft: Failed to contact a75e2fc6-3d9c-d681-67ad-f6786fe9e276 in 7.411204075s 2021/06/17 11:41:35 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:41:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.224.74:47766->10.242.71.138:8301: i/o timeout 2021/06/17 11:41:35 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/17 11:41:36 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:43784->10.242.71.138:8300: i/o timeout 2021/06/17 11:41:37 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:41:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.224.74:47778->10.242.71.138:8301: i/o timeout 2021/06/17 11:41:37 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/17 11:41:38 [WARN] raft: Failed to contact a75e2fc6-3d9c-d681-67ad-f6786fe9e276 in 2.50019198s 2021/06/17 11:41:39 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:41:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.224.74:47802->10.242.71.138:8301: i/o timeout 2021/06/17 11:41:39 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/17 11:41:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.224.74:52494->10.242.71.138:8302: i/o timeout 2021/06/17 11:41:40 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/06/17 11:41:41 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:41:41 [WARN] raft: Failed to contact a75e2fc6-3d9c-d681-67ad-f6786fe9e276 in 4.927607327s 2021/06/17 11:41:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.224.74:47846->10.242.71.138:8301: i/o timeout 2021/06/17 11:41:42 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/17 11:41:43 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:41:43 [WARN] raft: Failed to contact a75e2fc6-3d9c-d681-67ad-f6786fe9e276 in 7.400534982s 2021/06/17 11:41:43 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations) 2021/06/17 11:41:43 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.71.138 2021/06/17 11:41:43 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.71.138:8300) (DC: dc1) 2021/06/17 11:41:43 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical 2021/06/17 11:41:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.224.74:47862->10.242.71.138:8301: i/o timeout 2021/06/17 11:41:44 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/17 11:41:45 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:41:46 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:51572->10.242.71.138:8300: i/o timeout 2021/06/17 11:41:47 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:41:47 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:41:47 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: read tcp 10.242.224.74:43888->10.242.71.138:8300: i/o timeout 2021/06/17 11:41:48 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:41:49 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:41:51 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:41:53 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:41:55 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:41:55 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF 2021/06/17 11:41:55 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-tm7mq 10.242.71.136 2021/06/17 11:41:55 [INFO] consul: member 'dev-consul-7467bf7b46-tm7mq' joined, marking health alive 2021/06/17 11:41:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.224.74:52694->10.242.71.138:8302: i/o timeout 2021/06/17 11:41:55 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/06/17 11:41:57 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:43986->10.242.71.138:8300: i/o timeout 2021/06/17 11:41:57 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:41:58 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: read tcp 10.242.224.74:43996->10.242.71.138:8300: i/o timeout 2021/06/17 11:41:58 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:41:58 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:41:58 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:41:59 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:41:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.224.74:33832->10.242.71.136:8301: i/o timeout 2021/06/17 11:41:59 [INFO] memberlist: Suspect dev-consul-7467bf7b46-tm7mq has failed, no acks received 2021/06/17 11:42:01 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.224.74:33854->10.242.71.136:8301: i/o timeout 2021/06/17 11:42:01 [INFO] memberlist: Suspect dev-consul-7467bf7b46-tm7mq has failed, no acks received 2021/06/17 11:42:03 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:03 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:42:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.224.74:33890->10.242.71.136:8301: i/o timeout 2021/06/17 11:42:04 [INFO] memberlist: Suspect dev-consul-7467bf7b46-tm7mq has failed, no acks received 2021/06/17 11:42:05 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:05 [WARN] raft: Failed to contact a75e2fc6-3d9c-d681-67ad-f6786fe9e276 in 2.500338695s 2021/06/17 11:42:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.224.74:52740->10.242.71.138:8302: i/o timeout 2021/06/17 11:42:05 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/06/17 11:42:06 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/06/17 11:42:06 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.71.138 2021/06/17 11:42:06 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan" 2021/06/17 11:42:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.224.74:33898->10.242.71.136:8301: i/o timeout 2021/06/17 11:42:06 [INFO] memberlist: Suspect dev-consul-7467bf7b46-tm7mq has failed, no acks received 2021/06/17 11:42:07 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:44082->10.242.71.138:8300: i/o timeout 2021/06/17 11:42:07 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:42:07 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:07 [WARN] raft: Failed to contact a75e2fc6-3d9c-d681-67ad-f6786fe9e276 in 4.983526001s 2021/06/17 11:42:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.224.74:33928->10.242.71.136:8301: i/o timeout 2021/06/17 11:42:08 [INFO] memberlist: Suspect dev-consul-7467bf7b46-tm7mq has failed, no acks received 2021/06/17 11:42:09 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:09 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-tm7mq 10.242.71.136 2021/06/17 11:42:09 [INFO] consul: member 'dev-consul-7467bf7b46-tm7mq' failed, marking health critical 2021/06/17 11:42:10 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/17 11:42:10 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.71.138:60176 2021/06/17 11:42:10 [WARN] raft: Failed to contact a75e2fc6-3d9c-d681-67ad-f6786fe9e276 in 7.456383417s 2021/06/17 11:42:11 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:12 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:42:13 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:14 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:42:15 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:15 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:42:16 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:42:17 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:44188->10.242.71.138:8300: i/o timeout 2021/06/17 11:42:17 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:17 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:42:18 [WARN] raft: Failed to contact a75e2fc6-3d9c-d681-67ad-f6786fe9e276 in 2.500139831s 2021/06/17 11:42:19 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:20 [WARN] raft: Failed to contact a75e2fc6-3d9c-d681-67ad-f6786fe9e276 in 4.944073575s 2021/06/17 11:42:21 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:23 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:23 [WARN] raft: Failed to contact a75e2fc6-3d9c-d681-67ad-f6786fe9e276 in 7.420434629s 2021/06/17 11:42:25 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:25 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-tm7mq 10.242.71.136:8301 2021/06/17 11:42:25 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.71.138:8302 2021/06/17 11:42:26 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: read tcp 10.242.224.74:44102->10.242.71.138:8300: i/o timeout 2021/06/17 11:42:26 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:42:27 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:27 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:44286->10.242.71.138:8300: i/o timeout 2021/06/17 11:42:27 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:42:29 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:31 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:33 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:35 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:36 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: read tcp 10.242.224.74:44382->10.242.71.138:8300: i/o timeout 2021/06/17 11:42:37 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:37 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/17 11:42:37 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF 2021/06/17 11:42:37 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:44386->10.242.71.138:8300: i/o timeout 2021/06/17 11:42:37 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:42:37 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:42:40 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:42:41 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:43 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:45 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:47 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:47 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:44478->10.242.71.138:8300: i/o timeout 2021/06/17 11:42:47 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: read tcp 10.242.224.74:44480->10.242.71.138:8300: i/o timeout 2021/06/17 11:42:48 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:42:48 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:42:49 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:51 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:53 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:55 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:57 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:58 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:44582->10.242.71.138:8300: i/o timeout 2021/06/17 11:42:58 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: read tcp 10.242.224.74:44586->10.242.71.138:8300: i/o timeout 2021/06/17 11:42:59 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:42:59 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:42:59 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:43:01 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:03 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:05 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:05 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.71.138:8301 2021/06/17 11:43:07 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:09 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:09 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: read tcp 10.242.224.74:44682->10.242.71.138:8300: i/o timeout 2021/06/17 11:43:09 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:44688->10.242.71.138:8300: i/o timeout 2021/06/17 11:43:10 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:43:11 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:12 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:43:13 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:15 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:17 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:19 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:19 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/17 11:43:19 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF 2021/06/17 11:43:20 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: read tcp 10.242.224.74:44798->10.242.71.138:8300: i/o timeout 2021/06/17 11:43:21 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:43:22 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:43:22 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:43:22 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:44818->10.242.71.138:8300: i/o timeout 2021/06/17 11:43:23 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:25 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:25 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.71.138:8302 2021/06/17 11:43:27 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:27 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:43:28 [ERR] memberlist: failed to receive: read tcp 10.242.224.74:8301->10.242.71.138:52452: i/o timeout from=10.242.71.138:52452 2021/06/17 11:43:28 [ERR] memberlist: Failed to send error: write tcp 10.242.224.74:8301->10.242.71.138:52452: i/o timeout from=10.242.71.138:52452 2021/06/17 11:43:29 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:31 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:31 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: read tcp 10.242.224.74:44902->10.242.71.138:8300: i/o timeout 2021/06/17 11:43:32 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:43:33 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:35 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:37 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:37 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:44960->10.242.71.138:8300: i/o timeout 2021/06/17 11:43:39 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:41 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:42 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: read tcp 10.242.224.74:45012->10.242.71.138:8300: i/o timeout 2021/06/17 11:43:43 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:43 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:43:45 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:45 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.71.138:8301 2021/06/17 11:43:47 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:47 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:43:49 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:51 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:53 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:53 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: read tcp 10.242.224.74:45110->10.242.71.138:8300: i/o timeout 2021/06/17 11:43:55 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:55 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:43:57 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:43:57 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:45150->10.242.71.138:8300: i/o timeout 2021/06/17 11:43:59 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:01 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:01 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/17 11:44:01 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF 2021/06/17 11:44:04 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:44:04 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:44:04 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/06/17 11:44:05 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:05 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: read tcp 10.242.224.74:45224->10.242.71.138:8300: i/o timeout 2021/06/17 11:44:07 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:07 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:44:09 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:09 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:44:11 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:13 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:15 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:17 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:17 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:45336->10.242.71.138:8300: i/o timeout 2021/06/17 11:44:19 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:19 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: read tcp 10.242.224.74:45350->10.242.71.138:8300: i/o timeout 2021/06/17 11:44:21 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:23 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:24 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:44:25 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:25 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.71.138:8302 2021/06/17 11:44:25 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.71.138:8301 2021/06/17 11:44:27 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:27 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:44:29 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:31 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:33 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:34 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: read tcp 10.242.224.74:45504->10.242.71.138:8300: i/o timeout 2021/06/17 11:44:35 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:37 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:37 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:45532->10.242.71.138:8300: i/o timeout 2021/06/17 11:44:39 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:41 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:43 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:43 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/17 11:44:43 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF 2021/06/17 11:44:45 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:44:46 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:44:46 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:44:47 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:48 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:44:49 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:51 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:53 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:55 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:55 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: read tcp 10.242.224.74:45704->10.242.71.138:8300: i/o timeout 2021/06/17 11:44:57 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:44:58 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:45722->10.242.71.138:8300: i/o timeout 2021/06/17 11:44:59 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:01 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:03 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:05 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:05 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.71.138:8301 2021/06/17 11:45:07 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:45:07 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:08 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:45:09 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:11 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:13 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:15 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:17 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: read tcp 10.242.224.74:45898->10.242.71.138:8300: i/o timeout 2021/06/17 11:45:17 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:18 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:45908->10.242.71.138:8300: i/o timeout 2021/06/17 11:45:19 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:21 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:23 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:25 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:25 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/17 11:45:25 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF 2021/06/17 11:45:27 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:45:28 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:45:28 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:45:28 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/06/17 11:45:28 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:45:29 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:31 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:33 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:35 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:37 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:37 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: read tcp 10.242.224.74:46092->10.242.71.138:8300: i/o timeout 2021/06/17 11:45:38 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:46100->10.242.71.138:8300: i/o timeout 2021/06/17 11:45:39 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:41 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:43 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:45 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:45 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-tm7mq 10.242.71.136:8301 2021/06/17 11:45:47 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:48 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:45:49 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:49 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:45:51 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:53 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:55 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:55 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.71.138:8302 2021/06/17 11:45:57 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:58 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:46290->10.242.71.138:8300: i/o timeout 2021/06/17 11:45:59 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:45:59 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: read tcp 10.242.224.74:46292->10.242.71.138:8300: i/o timeout 2021/06/17 11:46:01 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:03 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:05 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:07 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:07 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/17 11:46:07 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF 2021/06/17 11:46:09 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:46:10 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:46:10 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:46:11 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:13 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:15 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:17 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:19 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:19 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:46478->10.242.71.138:8300: i/o timeout 2021/06/17 11:46:20 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: read tcp 10.242.224.74:46494->10.242.71.138:8300: i/o timeout 2021/06/17 11:46:21 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:23 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:25 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:25 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-tm7mq 10.242.71.136:8301 2021/06/17 11:46:27 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:29 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:29 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:46:30 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:46:31 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:33 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:35 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:37 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:39 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:39 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:46672->10.242.71.138:8300: i/o timeout 2021/06/17 11:46:40 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: read tcp 10.242.224.74:46694->10.242.71.138:8300: i/o timeout 2021/06/17 11:46:41 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:43 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:45 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:47 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:49 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:49 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/17 11:46:49 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF 2021/06/17 11:46:49 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:46:52 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:46:52 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:46:52 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:46:53 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:55 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:55 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.71.138:8302 2021/06/17 11:46:57 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:59 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:46:59 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: read tcp 10.242.224.74:46862->10.242.71.138:8300: i/o timeout 2021/06/17 11:47:01 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:47:02 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: read tcp 10.242.224.74:46896->10.242.71.138:8300: i/o timeout 2021/06/17 11:47:03 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:47:05 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:47:05 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-tm7mq 10.242.71.136:8301 2021/06/17 11:47:07 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:47:07 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.71.138 2021/06/17 11:47:07 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan" 2021/06/17 11:47:07 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.71.138 2021/06/17 11:47:07 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.71.138:8300) (DC: dc1) 2021/06/17 11:47:07 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-tm7mq 10.242.71.136 2021/06/17 11:47:07 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive 2021/06/17 11:47:07 [INFO] consul: member 'dev-consul-7467bf7b46-tm7mq' joined, marking health alive 2021/06/17 11:47:09 [INFO] raft: pipelining replication to peer {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300} 2021/06/17 11:47:19 [INFO] raft: aborting pipeline replication to peer {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300} 2021/06/17 11:47:19 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: EOF 2021/06/17 11:47:19 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: dial tcp ->10.242.71.138:8300: getsockopt: connection refused 2021/06/17 11:47:19 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: dial tcp ->10.242.71.138:8300: getsockopt: connection refused 2021/06/17 11:47:20 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: dial tcp ->10.242.71.138:8300: getsockopt: connection refused 2021/06/17 11:47:20 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: dial tcp ->10.242.71.138:8300: getsockopt: connection refused 2021/06/17 11:47:20 [INFO] serf: EventMemberLeave: dev-consul-7467bf7b46-tm7mq 10.242.71.136 2021/06/17 11:47:20 [INFO] consul: member 'dev-consul-7467bf7b46-tm7mq' left, deregistering 2021/06/17 11:47:20 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: dial tcp ->10.242.71.138:8300: getsockopt: connection refused 2021/06/17 11:47:20 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: dial tcp ->10.242.71.138:8300: getsockopt: connection refused 2021/06/17 11:47:20 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: dial tcp ->10.242.71.138:8300: getsockopt: connection refused 2021/06/17 11:47:20 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: dial tcp ->10.242.71.138:8300: getsockopt: connection refused 2021/06/17 11:47:20 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: dial tcp ->10.242.71.138:8300: getsockopt: connection refused 2021/06/17 11:47:21 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: stream closed 2021/06/17 11:47:21 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: dial tcp ->10.242.71.138:8300: getsockopt: connection refused 2021/06/17 11:47:21 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: dial tcp ->10.242.71.138:8300: getsockopt: connection refused 2021/06/17 11:47:21 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/17 11:47:22 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:47:22 [WARN] raft: Failed to contact a75e2fc6-3d9c-d681-67ad-f6786fe9e276 in 2.500178181s 2021/06/17 11:47:22 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: dial tcp ->10.242.71.138:8300: getsockopt: connection refused 2021/06/17 11:47:22 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: dial tcp ->10.242.71.138:8300: getsockopt: connection refused 2021/06/17 11:47:23 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.71.138:8300: getsockopt: connection refused 2021/06/17 11:47:23 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: dial tcp ->10.242.71.138:8300: getsockopt: connection refused 2021/06/17 11:47:23 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/17 11:47:23 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: dial tcp ->10.242.71.138:8300: getsockopt: connection refused 2021/06/17 11:47:24 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:47:24 [WARN] raft: Failed to contact a75e2fc6-3d9c-d681-67ad-f6786fe9e276 in 4.986461966s 2021/06/17 11:47:24 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: dial tcp ->10.242.71.138:8300: getsockopt: connection refused 2021/06/17 11:47:25 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.71.138:8300: getsockopt: connection refused 2021/06/17 11:47:25 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: dial tcp ->10.242.71.138:8300: getsockopt: connection refused 2021/06/17 11:47:25 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/06/17 11:47:25 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/17 11:47:25 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: dial tcp ->10.242.71.138:8300: getsockopt: connection refused 2021/06/17 11:47:26 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:47:26 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: dial tcp ->10.242.71.138:8300: getsockopt: connection refused 2021/06/17 11:47:27 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.71.138:8300: getsockopt: connection refused 2021/06/17 11:47:27 [WARN] raft: Failed to contact a75e2fc6-3d9c-d681-67ad-f6786fe9e276 in 7.48162137s 2021/06/17 11:47:28 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:47:30 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:47:30 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:47:31 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:47:33 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations) 2021/06/17 11:47:33 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.71.138 2021/06/17 11:47:33 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.71.138:8300) (DC: dc1) 2021/06/17 11:47:33 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical 2021/06/17 11:47:33 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:47:33 [INFO] consul: member 'dev-consul-7467bf7b46-tm7mq' left, deregistering 2021/06/17 11:47:35 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:47:35 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-rgrsk 10.242.2.83 2021/06/17 11:47:35 [INFO] consul: member 'dev-consul-7467bf7b46-rgrsk' joined, marking health alive 2021/06/17 11:47:37 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:47:37 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.71.138:8301 2021/06/17 11:47:38 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/17 11:47:38 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: dial tcp ->10.242.71.138:8300: i/o timeout 2021/06/17 11:47:39 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:47:39 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.71.138:8300: i/o timeout 2021/06/17 11:47:40 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: dial tcp ->10.242.71.138:8300: i/o timeout 2021/06/17 11:47:42 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:47:42 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:47:42 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:47:43 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:47:45 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:47:47 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:47:49 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:47:50 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:47:51 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:47:51 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.71.138:8300: i/o timeout 2021/06/17 11:47:52 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: dial tcp ->10.242.71.138:8300: i/o timeout 2021/06/17 11:47:54 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:47:54 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:47:55 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:47:55 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/06/17 11:47:55 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.71.138 2021/06/17 11:47:55 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan" 2021/06/17 11:47:57 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:47:58 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:47:59 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:48:00 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: dial tcp ->10.242.71.138:8300: i/o timeout 2021/06/17 11:48:01 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:48:03 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:48:03 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.71.138:8300: i/o timeout 2021/06/17 11:48:03 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/06/17 11:48:06 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/17 11:48:06 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/17 11:48:07 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/06/17 11:48:07 [ERR] memberlist: Conflicting address for dev-consul-server-0.dc1. Mine: 10.242.71.138:8302 Theirs: 10.242.2.87:8302 2021/06/17 11:48:07 [WARN] serf: Name conflict for 'dev-consul-server-0.dc1' both 10.242.71.138:8302 and 10.242.2.87:8302 are claiming 2021/06/17 11:48:07 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.2.87 2021/06/17 11:48:07 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.2.87:8300) (DC: dc1) 2021/06/17 11:48:07 [INFO] raft: Updating configuration with AddNonvoter (2c487dc7-8737-d351-ddd4-91445c5b5441, 10.242.2.87:8300) to [{Suffrage:Voter ID:ebda65a7-b47f-c28e-e8cd-05aadf21320c Address:10.242.224.74:8300} {Suffrage:Voter ID:a75e2fc6-3d9c-d681-67ad-f6786fe9e276 Address:10.242.71.138:8300} {Suffrage:Voter ID:080c2ccf-91d8-bfc6-205b-266925ed7eaa Address:10.242.105.8:8300} {Suffrage:Nonvoter ID:2c487dc7-8737-d351-ddd4-91445c5b5441 Address:10.242.2.87:8300}] 2021/06/17 11:48:07 [INFO] raft: Added peer 2c487dc7-8737-d351-ddd4-91445c5b5441, starting replication 2021/06/17 11:48:07 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive 2021/06/17 11:48:07 [WARN] raft: AppendEntries to {Nonvoter 2c487dc7-8737-d351-ddd4-91445c5b5441 10.242.2.87:8300} rejected, sending older logs (next: 1) 2021/06/17 11:48:07 [INFO] raft: pipelining replication to peer {Nonvoter 2c487dc7-8737-d351-ddd4-91445c5b5441 10.242.2.87:8300} 2021/06/17 11:48:07 [ERR] memberlist: Conflicting address for dev-consul-server-0.dc1. Mine: 10.242.71.138:8302 Theirs: 10.242.2.87:8302 2021/06/17 11:48:07 [WARN] serf: Name conflict for 'dev-consul-server-0.dc1' both 10.242.71.138:8302 and 10.242.2.87:8302 are claiming 2021/06/17 11:48:07 [ERR] memberlist: Conflicting address for dev-consul-server-0.dc1. Mine: 10.242.71.138:8302 Theirs: 10.242.2.87:8302 2021/06/17 11:48:07 [WARN] serf: Name conflict for 'dev-consul-server-0.dc1' both 10.242.71.138:8302 and 10.242.2.87:8302 are claiming 2021/06/17 11:48:08 [ERR] memberlist: Conflicting address for dev-consul-server-0.dc1. Mine: 10.242.71.138:8302 Theirs: 10.242.2.87:8302 2021/06/17 11:48:08 [WARN] serf: Name conflict for 'dev-consul-server-0.dc1' both 10.242.71.138:8302 and 10.242.2.87:8302 are claiming 2021/06/17 11:48:08 [ERR] memberlist: Conflicting address for dev-consul-server-0.dc1. Mine: 10.242.71.138:8302 Theirs: 10.242.2.87:8302 2021/06/17 11:48:08 [WARN] serf: Name conflict for 'dev-consul-server-0.dc1' both 10.242.71.138:8302 and 10.242.2.87:8302 are claiming 2021/06/17 11:48:08 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: dial tcp ->10.242.71.138:8300: i/o timeout 2021/06/17 11:48:08 [ERR] memberlist: Conflicting address for dev-consul-server-0.dc1. Mine: 10.242.71.138:8302 Theirs: 10.242.2.87:8302 2021/06/17 11:48:08 [WARN] serf: Name conflict for 'dev-consul-server-0.dc1' both 10.242.71.138:8302 and 10.242.2.87:8302 are claiming 2021/06/17 11:48:08 [ERR] memberlist: Conflicting address for dev-consul-server-0.dc1. Mine: 10.242.71.138:8302 Theirs: 10.242.2.87:8302 2021/06/17 11:48:08 [WARN] serf: Name conflict for 'dev-consul-server-0.dc1' both 10.242.71.138:8302 and 10.242.2.87:8302 are claiming 2021/06/17 11:48:11 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:48:15 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.71.138:8300: i/o timeout 2021/06/17 11:48:19 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:48:21 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: dial tcp ->10.242.71.138:8300: i/o timeout 2021/06/17 11:48:23 [INFO] autopilot: Promoting Server (ID: "2c487dc7-8737-d351-ddd4-91445c5b5441" Address: "10.242.2.87:8300") to voter 2021/06/17 11:48:23 [INFO] raft: Updating configuration with AddStaging (2c487dc7-8737-d351-ddd4-91445c5b5441, 10.242.2.87:8300) to [{Suffrage:Voter ID:ebda65a7-b47f-c28e-e8cd-05aadf21320c Address:10.242.224.74:8300} {Suffrage:Voter ID:a75e2fc6-3d9c-d681-67ad-f6786fe9e276 Address:10.242.71.138:8300} {Suffrage:Voter ID:080c2ccf-91d8-bfc6-205b-266925ed7eaa Address:10.242.105.8:8300} {Suffrage:Voter ID:2c487dc7-8737-d351-ddd4-91445c5b5441 Address:10.242.2.87:8300}] 2021/06/17 11:48:23 [INFO] autopilot: Attempting removal of stale Server (ID: "a75e2fc6-3d9c-d681-67ad-f6786fe9e276" Address: "10.242.71.138:8300") 2021/06/17 11:48:23 [INFO] raft: Updating configuration with RemoveServer (a75e2fc6-3d9c-d681-67ad-f6786fe9e276, ) to [{Suffrage:Voter ID:ebda65a7-b47f-c28e-e8cd-05aadf21320c Address:10.242.224.74:8300} {Suffrage:Voter ID:080c2ccf-91d8-bfc6-205b-266925ed7eaa Address:10.242.105.8:8300} {Suffrage:Voter ID:2c487dc7-8737-d351-ddd4-91445c5b5441 Address:10.242.2.87:8300}] 2021/06/17 11:48:23 [INFO] raft: Removed peer a75e2fc6-3d9c-d681-67ad-f6786fe9e276, stopping replication after 867 2021/06/17 11:48:29 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: dial tcp ->10.242.71.138:8300: i/o timeout 2021/06/17 11:48:31 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:48:40 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:48:41 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: dial tcp ->10.242.71.138:8300: i/o timeout 2021/06/17 11:48:50 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: dial tcp ->10.242.71.138:8300: i/o timeout 2021/06/17 11:48:51 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:49:01 [WARN] Unable to get address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276, using fallback address 10.242.71.138:8300: Could not find address for server id a75e2fc6-3d9c-d681-67ad-f6786fe9e276 2021/06/17 11:49:01 [ERR] raft: Failed to AppendEntries to {Voter a75e2fc6-3d9c-d681-67ad-f6786fe9e276 10.242.71.138:8300}: dial tcp ->10.242.71.138:8300: i/o timeout 2021/06/17 11:49:11 [ERR] raft: Failed to heartbeat to 10.242.71.138:8300: dial tcp ->10.242.71.138:8300: i/o timeout 2021/06/17 11:50:03 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.2.87 2021/06/17 11:50:03 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"