By type
+ CONSUL_BIND= + [ -n ] + CONSUL_CLIENT= + [ -n ] + CONSUL_DATA_DIR=/consul/data + CONSUL_CONFIG_DIR=/consul/config + [ -n ] + [ a = - ] + [ agent = agent ] + shift + set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui + [ consul = consul ] + stat -c %u /consul/data + id -u consul + [ 100 != 100 ] + stat -c %u /consul/config + id -u consul + [ 100 != 100 ] + [ ! -z ] + exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui bootstrap_expect > 0: expecting 3 servers ==> Starting Consul agent... ==> Consul agent running! Version: 'v1.0.6' Node ID: '5fa9f731-8cb6-e040-8d5f-1f0ab18b143a' Node name: 'dev-consul-server-0' Datacenter: 'dc1' (Segment: '
') Server: true (Bootstrap: false) Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600) Cluster Addr: 10.242.7.75 (LAN: 8301, WAN: 8302) Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false ==> Log data will now stream in as it occurs: 2021/06/05 10:40:01 [INFO] raft: Initial configuration (index=0): [] 2021/06/05 10:40:01 [INFO] raft: Node at 10.242.7.75:8300 [Follower] entering Follower state (Leader: "") 2021/06/05 10:40:01 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.7.75 2021/06/05 10:40:01 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.7.75 2021/06/05 10:40:01 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan" 2021/06/05 10:40:01 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.7.75:8300) (DC: dc1) 2021/06/05 10:40:01 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp) 2021/06/05 10:40:01 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp) 2021/06/05 10:40:01 [INFO] agent: Started HTTP server on [::]:8500 (tcp) 2021/06/05 10:40:01 [INFO] agent: started state syncer 2021/06/05 10:40:01 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer 2021/06/05 10:40:01 [INFO] agent: Joining LAN cluster... 2021/06/05 10:40:01 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server] 2021/06/05 10:40:01 [WARN] memberlist: Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host 2021/06/05 10:40:01 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host 2021/06/05 10:40:02 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host 2021/06/05 10:40:02 [INFO] agent: (LAN) joined: 0 Err: 3 error(s) occurred: * Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host * Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host * Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host 2021/06/05 10:40:02 [WARN] agent: Join LAN failed: , retrying in 30s 2021/06/05 10:40:09 [ERR] agent: failed to sync remote state: No cluster leader 2021/06/05 10:40:10 [WARN] raft: no known peers, aborting election ==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=a15c3ab4-1aac-3a88-302f-ca1f320a66b5&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 2021/06/05 10:40:24 [ERR] agent: Coordinate update error: No cluster leader 2021/06/05 10:40:31 [ERR] agent: failed to sync remote state: No cluster leader 2021/06/05 10:40:31 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-5tc7t 10.242.7.74 2021/06/05 10:40:32 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server] 2021/06/05 10:40:32 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host 2021/06/05 10:40:32 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host 2021/06/05 10:40:32 [INFO] agent: (LAN) joined: 1 Err: 2021/06/05 10:40:32 [INFO] agent: Join LAN completed. Synced with 1 initial agents 2021/06/05 10:40:37 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.37.70 2021/06/05 10:40:37 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.37.70:8300) (DC: dc1) 2021/06/05 10:40:37 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.37.70 2021/06/05 10:40:37 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan" 2021/06/05 10:40:53 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 10:40:53 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 10:40:53 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.242.37.70:8300,10.242.204.71:8300,10.242.7.75:8300 2021/06/05 10:40:53 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 10:40:53 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/05 10:40:54 [DEBUG] raft-net: 10.242.7.75:8300 accepted connection from: 10.242.204.71:42352 2021/06/05 10:40:54 [DEBUG] raft-net: 10.242.7.75:8300 accepted connection from: 10.242.204.71:42354 2021/06/05 10:40:54 [INFO] consul: New leader elected: dev-consul-server-1 2021/06/05 10:40:55 [INFO] agent: Synced node info 2021/06/05 11:45:24 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 11:45:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:56064->10.242.204.71:8301: i/o timeout 2021/06/05 11:45:24 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:45:26 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 11:45:26 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 11:45:26 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:45:28 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 11:45:28 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:46:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:56946->10.242.204.71:8301: i/o timeout 2021/06/05 11:46:11 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:46:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:56994->10.242.204.71:8301: i/o timeout 2021/06/05 11:46:14 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:46:14 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 11:46:14 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:46:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:51876->10.242.204.71:8302: i/o timeout 2021/06/05 11:46:16 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:46:19 [WARN] raft: Heartbeat timeout from "10.242.204.71:8300" reached, starting election 2021/06/05 11:46:19 [INFO] raft: Node at 10.242.7.75:8300 [Candidate] entering Candidate state in term 3 2021/06/05 11:46:19 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:46:22 [DEBUG] raft-net: 10.242.7.75:8300 accepted connection from: 10.242.37.70:47560 2021/06/05 11:46:22 [INFO] raft: Duplicate RequestVote for same term: 3 2021/06/05 11:46:25 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.37.85:60164 2021/06/05 11:46:25 [WARN] raft: Election timeout reached, restarting election 2021/06/05 11:46:25 [INFO] raft: Node at 10.242.7.75:8300 [Candidate] entering Candidate state in term 4 2021/06/05 11:46:25 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:46:25 [INFO] raft: Election won. Tally: 2 2021/06/05 11:46:25 [INFO] raft: Node at 10.242.7.75:8300 [Leader] entering Leader state 2021/06/05 11:46:25 [INFO] raft: Added peer d942fbe0-30f7-c6b7-2f54-9c43c9c150dd, starting replication 2021/06/05 11:46:25 [INFO] consul: cluster leadership acquired 2021/06/05 11:46:25 [INFO] raft: Added peer 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, starting replication 2021/06/05 11:46:25 [INFO] consul: New leader elected: dev-consul-server-0 2021/06/05 11:46:25 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:46:25 [INFO] raft: pipelining replication to peer {Voter d942fbe0-30f7-c6b7-2f54-9c43c9c150dd 10.242.37.70:8300} 2021/06/05 11:46:25 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical 2021/06/05 11:46:26 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:46:27 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.501935955s 2021/06/05 11:46:28 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:46:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:46:29 [ERR] raft: Failed to make RequestVote RPC to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:41686->10.242.204.71:8300: i/o timeout 2021/06/05 11:46:30 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 4.940464391s 2021/06/05 11:46:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:46:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:52056->10.242.204.71:8302: i/o timeout 2021/06/05 11:46:31 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:46:31 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.204.71:8301 2021/06/05 11:46:32 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 7.381860485s 2021/06/05 11:46:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:46:34 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/05 11:46:34 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: rpc error making call: EOF from=10.242.79.8:33246 2021/06/05 11:46:34 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/06/05 11:46:35 [ERR] raft: Failed to make RequestVote RPC to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:41774->10.242.204.71:8300: i/o timeout 2021/06/05 11:46:35 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:41776->10.242.204.71:8300: i/o timeout 2021/06/05 11:46:35 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:46:36 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:41808->10.242.204.71:8300: i/o timeout 2021/06/05 11:46:36 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/05 11:46:36 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.204.71:42330 2021/06/05 11:46:36 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:46:36 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/05 11:46:36 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:46:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:46:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:46:40 [ERR] memberlist: Push/Pull with dev-consul-server-1.dc1 failed: read tcp 10.242.7.75:51798->10.242.204.71:8302: i/o timeout 2021/06/05 11:46:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:46:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:52368->10.242.204.71:8302: i/o timeout 2021/06/05 11:46:41 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:46:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:46:43 [ERR] raft-net: Failed to decode incoming command: read tcp 10.242.7.75:8300->10.242.204.71:42354: read: connection reset by peer 2021/06/05 11:46:44 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 11:46:44 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:46:44 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/06/05 11:46:44 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/05 11:46:45 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:41972->10.242.204.71:8300: i/o timeout 2021/06/05 11:46:46 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:46:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:57626->10.242.204.71:8301: i/o timeout 2021/06/05 11:46:46 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:46:46 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500269591s 2021/06/05 11:46:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:46:49 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 4.946116226s 2021/06/05 11:46:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:46:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:57662->10.242.204.71:8301: i/o timeout 2021/06/05 11:46:49 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:46:50 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 11:46:50 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 11:46:50 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:46:50 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical 2021/06/05 11:46:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:46:51 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 7.413692374s 2021/06/05 11:46:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:46:54 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:42016->10.242.204.71:8300: i/o timeout 2021/06/05 11:46:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:46:55 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:42162->10.242.204.71:8300: i/o timeout 2021/06/05 11:46:55 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:46:55 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:46:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:52566->10.242.204.71:8302: i/o timeout 2021/06/05 11:46:56 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:46:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:46:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:05 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:42350->10.242.204.71:8300: i/o timeout 2021/06/05 11:47:05 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:47:05 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:42352->10.242.204.71:8300: i/o timeout 2021/06/05 11:47:05 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:47:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:52942->10.242.204.71:8302: i/o timeout 2021/06/05 11:47:11 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:47:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:15 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/05 11:47:15 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/06/05 11:47:15 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:42538->10.242.204.71:8300: i/o timeout 2021/06/05 11:47:15 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:47:15 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:42566->10.242.204.71:8300: i/o timeout 2021/06/05 11:47:16 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:47:18 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:47:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:53126->10.242.204.71:8302: i/o timeout 2021/06/05 11:47:21 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:47:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:24 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/05 11:47:24 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.204.71:45820 2021/06/05 11:47:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:25 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:42732->10.242.204.71:8300: i/o timeout 2021/06/05 11:47:25 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:47:26 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:42764->10.242.204.71:8300: i/o timeout 2021/06/05 11:47:26 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/06/05 11:47:26 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 11:47:26 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/05 11:47:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:27 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:47:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:53326->10.242.204.71:8302: i/o timeout 2021/06/05 11:47:31 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:47:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:35 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:42934->10.242.204.71:8300: i/o timeout 2021/06/05 11:47:35 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:47:36 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:47:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:37 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:47:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:39 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500123716s 2021/06/05 11:47:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:41 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.204.71:8301 2021/06/05 11:47:41 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 4.958013072s 2021/06/05 11:47:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:44 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 7.445915247s 2021/06/05 11:47:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:45 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:42988->10.242.204.71:8300: i/o timeout 2021/06/05 11:47:46 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:47:46 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:47:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:48 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500511159s 2021/06/05 11:47:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:51 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 4.930541646s 2021/06/05 11:47:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:47:53 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 7.429950655s 2021/06/05 11:47:53 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 11:47:53 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:47:53 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/06/05 11:47:53 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/05 11:47:53 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 11:47:53 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/05 11:47:53 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 11:47:54 [INFO] raft: pipelining replication to peer {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300} 2021/06/05 11:48:08 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:48:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/06/05 11:48:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:48:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:48:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:48:14 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured 2021/06/05 11:48:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:48:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:59276->10.242.204.71:8301: i/o timeout 2021/06/05 11:48:15 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:48:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:48:22 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:48:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:59416->10.242.204.71:8301: i/o timeout 2021/06/05 11:48:22 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:48:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:48:25 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500183016s 2021/06/05 11:48:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:48:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:54322->10.242.204.71:8302: i/o timeout 2021/06/05 11:48:26 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:48:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:59506->10.242.204.71:8301: i/o timeout 2021/06/05 11:48:26 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:48:26 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 11:48:26 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 11:48:26 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:48:26 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical 2021/06/05 11:48:27 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 4.999686961s 2021/06/05 11:48:28 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:48:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:48:30 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 7.497127748s 2021/06/05 11:48:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:48:32 [INFO] raft: aborting pipeline replication to peer {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300} 2021/06/05 11:48:32 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:48:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:48:33 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:43336->10.242.204.71:8300: i/o timeout 2021/06/05 11:48:33 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/05 11:48:33 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.204.71:46456 2021/06/05 11:48:34 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:48:34 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500654805s 2021/06/05 11:48:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:48:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:54446->10.242.204.71:8302: i/o timeout 2021/06/05 11:48:36 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:48:37 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 4.963817861s 2021/06/05 11:48:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:48:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:48:39 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 7.421777257s 2021/06/05 11:48:40 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 11:48:40 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:48:40 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/06/05 11:48:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:48:42 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:44184->10.242.204.71:8300: i/o timeout 2021/06/05 11:48:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:59802->10.242.204.71:8301: i/o timeout 2021/06/05 11:48:42 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:48:44 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:48:44 [INFO] raft: pipelining replication to peer {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300} 2021/06/05 11:48:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:48:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:59884->10.242.204.71:8301: i/o timeout 2021/06/05 11:48:46 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:48:48 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:48:48 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/05 11:48:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:48:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:59946->10.242.204.71:8301: i/o timeout 2021/06/05 11:48:49 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:48:49 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500313767s 2021/06/05 11:48:50 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 11:48:50 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 11:48:50 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:48:50 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical 2021/06/05 11:48:50 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 11:48:50 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:48:50 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/06/05 11:49:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:60194->10.242.204.71:8301: i/o timeout 2021/06/05 11:49:03 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:49:04 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:49:04 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/05 11:49:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:49:06 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500183686s 2021/06/05 11:49:06 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 11:49:08 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:49:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:60414->10.242.204.71:8301: i/o timeout 2021/06/05 11:49:15 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:49:16 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:49:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:49:17 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500199292s 2021/06/05 11:49:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:49:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:60504->10.242.204.71:8301: i/o timeout 2021/06/05 11:49:19 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:49:19 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 11:49:19 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 11:49:19 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:49:19 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical 2021/06/05 11:49:19 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 4.947234959s 2021/06/05 11:49:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:49:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:55384->10.242.204.71:8302: i/o timeout 2021/06/05 11:49:21 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:49:21 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.204.71:8301 2021/06/05 11:49:22 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 7.428893628s 2021/06/05 11:49:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:49:25 [INFO] raft: aborting pipeline replication to peer {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300} 2021/06/05 11:49:25 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:49:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:49:25 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:44376->10.242.204.71:8300: i/o timeout 2021/06/05 11:49:26 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:49:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:49:27 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.50024819s 2021/06/05 11:49:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:49:29 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 4.991233279s 2021/06/05 11:49:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:49:32 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 7.459184027s 2021/06/05 11:49:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:49:34 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:49:34 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 11:49:34 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:49:34 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/06/05 11:49:35 [WARN] raft: Rejecting vote request from 10.242.204.71:8300 since we have a leader: 10.242.7.75:8300 2021/06/05 11:49:35 [INFO] raft: pipelining replication to peer {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300} 2021/06/05 11:49:38 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:49:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:50:08 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:50:08 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/05 11:50:08 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500271054s 2021/06/05 11:50:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:33184->10.242.204.71:8301: i/o timeout 2021/06/05 11:50:08 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:50:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:50:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:33412->10.242.204.71:8301: i/o timeout 2021/06/05 11:50:21 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:50:22 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:50:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:50:24 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.50029985s 2021/06/05 11:50:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:50:25 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 11:50:25 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 11:50:25 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:50:25 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical 2021/06/05 11:50:27 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:50:28 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:50:28 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:50:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:50:30 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 11:50:30 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:50:30 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/06/05 11:50:32 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:50:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:33622->10.242.204.71:8301: i/o timeout 2021/06/05 11:50:32 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:50:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:50:34 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500140627s 2021/06/05 11:50:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:50:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:56752->10.242.204.71:8302: i/o timeout 2021/06/05 11:50:36 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:50:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:33690->10.242.204.71:8301: i/o timeout 2021/06/05 11:50:36 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:50:36 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 11:50:36 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 11:50:36 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:50:36 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical 2021/06/05 11:50:36 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 4.952367263s 2021/06/05 11:50:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:50:39 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 7.405136283s 2021/06/05 11:50:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:50:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:50:41 [INFO] raft: aborting pipeline replication to peer {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300} 2021/06/05 11:50:41 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:50:42 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:45206->10.242.204.71:8300: i/o timeout 2021/06/05 11:50:43 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:50:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:50:44 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.501106989s 2021/06/05 11:50:44 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/05 11:50:44 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/06/05 11:50:46 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:50:46 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/05 11:50:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:56948->10.242.204.71:8302: i/o timeout 2021/06/05 11:50:46 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:50:46 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 4.964579612s 2021/06/05 11:50:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:50:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:50:49 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 7.405805719s 2021/06/05 11:50:51 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/05 11:50:51 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.204.71:47350 2021/06/05 11:50:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:50:51 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:46606->10.242.204.71:8300: i/o timeout 2021/06/05 11:50:52 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:50:53 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:46622->10.242.204.71:8300: i/o timeout 2021/06/05 11:50:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:50:53 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:50:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:50:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:50:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:51:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:51:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:57138->10.242.204.71:8302: i/o timeout 2021/06/05 11:51:01 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:51:02 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:46802->10.242.204.71:8300: i/o timeout 2021/06/05 11:51:02 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:51:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:51:03 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:46820->10.242.204.71:8300: i/o timeout 2021/06/05 11:51:04 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:51:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:51:06 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/06/05 11:51:06 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 11:51:06 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/05 11:51:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:51:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:51:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:51:12 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:46996->10.242.204.71:8300: i/o timeout 2021/06/05 11:51:12 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:51:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:51:13 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 11:51:13 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:51:13 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/06/05 11:51:14 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 11:51:14 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/05 11:51:14 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/05 11:51:15 [WARN] raft: Rejecting vote request from 10.242.204.71:8300 since we have a leader: 10.242.7.75:8300 2021/06/05 11:51:15 [INFO] raft: pipelining replication to peer {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300} 2021/06/05 11:51:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:51:34 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:51:46 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:51:46 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/05 11:51:50 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:51:50 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.5003183s 2021/06/05 11:51:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:51:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:51:53 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 11:52:02 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:52:02 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/05 11:52:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:52:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:35312->10.242.204.71:8301: i/o timeout 2021/06/05 11:52:03 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:52:04 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500191419s 2021/06/05 11:52:05 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 11:52:10 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:52:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:52:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:52:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:35648->10.242.204.71:8301: i/o timeout 2021/06/05 11:52:21 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:52:22 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 11:52:22 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:52:22 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical 2021/06/05 11:52:23 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:52:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:52:24 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:52:24 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 11:52:24 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:52:24 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/06/05 11:52:24 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured 2021/06/05 11:52:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:52:26 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 11:52:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:35748->10.242.204.71:8301: i/o timeout 2021/06/05 11:52:26 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:52:30 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:52:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:35818->10.242.204.71:8301: i/o timeout 2021/06/05 11:52:30 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:52:30 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 11:52:30 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 11:52:30 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:52:30 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical 2021/06/05 11:52:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:52:31 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500249242s 2021/06/05 11:52:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:58926->10.242.204.71:8302: i/o timeout 2021/06/05 11:52:31 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:52:32 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:52:34 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:52:34 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500380193s 2021/06/05 11:52:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:52:36 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 4.955892609s 2021/06/05 11:52:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:52:38 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:52:39 [INFO] raft: aborting pipeline replication to peer {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300} 2021/06/05 11:52:39 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:52:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:52:40 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:52:41 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:52:42 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:52:43 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500211251s 2021/06/05 11:52:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:52:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:52:45 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 4.982458881s 2021/06/05 11:52:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:52:48 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 7.451264092s 2021/06/05 11:52:49 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:48794->10.242.204.71:8300: i/o timeout 2021/06/05 11:52:49 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:52:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:52:49 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 11:52:49 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:52:49 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/06/05 11:52:50 [WARN] raft: Rejecting vote request from 10.242.204.71:8300 since we have a leader: 10.242.7.75:8300 2021/06/05 11:52:50 [INFO] raft: pipelining replication to peer {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300} 2021/06/05 11:53:14 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:53:14 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/05 11:53:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:36630->10.242.204.71:8301: i/o timeout 2021/06/05 11:53:14 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:53:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:53:16 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500312034s 2021/06/05 11:53:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:53:18 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 4.986352131s 2021/06/05 11:53:18 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 11:53:18 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 11:53:18 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:53:18 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical 2021/06/05 11:53:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:53:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:36742->10.242.204.71:8301: i/o timeout 2021/06/05 11:53:19 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:53:21 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 7.473600017s 2021/06/05 11:53:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:53:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:59846->10.242.204.71:8302: i/o timeout 2021/06/05 11:53:22 [INFO] raft: aborting pipeline replication to peer {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300} 2021/06/05 11:53:22 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:53:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:53:24 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:53:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:53:26 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500275034s 2021/06/05 11:53:28 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:53:28 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 4.983312397s 2021/06/05 11:53:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:53:31 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 7.427772515s 2021/06/05 11:53:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:53:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:60036->10.242.204.71:8302: i/o timeout 2021/06/05 11:53:31 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:53:32 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 11:53:32 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:53:32 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/06/05 11:53:32 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:49596->10.242.204.71:8300: i/o timeout 2021/06/05 11:53:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:53:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:36992->10.242.204.71:8301: i/o timeout 2021/06/05 11:53:33 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:53:34 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:47020->10.242.204.71:8300: i/o timeout 2021/06/05 11:53:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:53:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:37024->10.242.204.71:8301: i/o timeout 2021/06/05 11:53:36 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:53:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:53:37 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 11:53:37 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 11:53:37 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:53:37 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical 2021/06/05 11:53:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:37100->10.242.204.71:8301: i/o timeout 2021/06/05 11:53:38 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:53:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:53:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:53:41 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured 2021/06/05 11:53:42 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:49782->10.242.204.71:8300: i/o timeout 2021/06/05 11:53:42 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:53:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:53:45 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:49820->10.242.204.71:8300: i/o timeout 2021/06/05 11:53:45 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:53:46 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/05 11:53:46 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:53:46 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:53:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:53:48 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500289375s 2021/06/05 11:53:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:53:50 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 4.962802313s 2021/06/05 11:53:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:53:52 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:49978->10.242.204.71:8300: i/o timeout 2021/06/05 11:53:52 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:53:53 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 7.397644497s 2021/06/05 11:53:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:53:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:53:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:60418->10.242.204.71:8302: i/o timeout 2021/06/05 11:53:56 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:53:56 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:50046->10.242.204.71:8300: i/o timeout 2021/06/05 11:53:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:53:57 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:53:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:00 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/05 11:54:00 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.204.71:49430 2021/06/05 11:54:00 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/05 11:54:00 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/06/05 11:54:02 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:54:02 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:50172->10.242.204.71:8300: i/o timeout 2021/06/05 11:54:02 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:54:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:03 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 11:54:04 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:54:05 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 11:54:05 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:54:05 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/06/05 11:54:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:05 [INFO] raft: pipelining replication to peer {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300} 2021/06/05 11:54:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:37656->10.242.204.71:8301: i/o timeout 2021/06/05 11:54:07 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:54:10 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:54:10 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 11:54:10 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 11:54:10 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:54:10 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical 2021/06/05 11:54:10 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured 2021/06/05 11:54:10 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:54:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:11 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured 2021/06/05 11:54:11 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:54:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:13 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:54:14 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:54:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:15 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:54:16 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:54:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:18 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 11:54:18 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:54:18 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/06/05 11:54:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:54:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:37894->10.242.204.71:8301: i/o timeout 2021/06/05 11:54:20 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:54:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:21 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500294275s 2021/06/05 11:54:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:37948->10.242.204.71:8301: i/o timeout 2021/06/05 11:54:23 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:54:24 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 4.977390832s 2021/06/05 11:54:24 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 11:54:24 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 11:54:24 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:54:24 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical 2021/06/05 11:54:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:32830->10.242.204.71:8302: i/o timeout 2021/06/05 11:54:26 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:54:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:37982->10.242.204.71:8301: i/o timeout 2021/06/05 11:54:26 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:54:26 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 7.441880045s 2021/06/05 11:54:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:29 [INFO] raft: aborting pipeline replication to peer {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300} 2021/06/05 11:54:29 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:54:29 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:50278->10.242.204.71:8300: i/o timeout 2021/06/05 11:54:30 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:54:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:31 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500175939s 2021/06/05 11:54:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:34 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 4.989113653s 2021/06/05 11:54:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:32952->10.242.204.71:8302: i/o timeout 2021/06/05 11:54:36 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:54:36 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 7.457127244s 2021/06/05 11:54:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:39 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:50880->10.242.204.71:8300: i/o timeout 2021/06/05 11:54:39 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:54:40 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:50900->10.242.204.71:8300: i/o timeout 2021/06/05 11:54:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:41 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/05 11:54:41 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/06/05 11:54:41 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:54:44 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:54:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:33208->10.242.204.71:8302: i/o timeout 2021/06/05 11:54:46 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:54:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:48 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/05 11:54:48 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.204.71:52026 2021/06/05 11:54:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:49 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:51058->10.242.204.71:8300: i/o timeout 2021/06/05 11:54:49 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:54:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:51 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:51100->10.242.204.71:8300: i/o timeout 2021/06/05 11:54:52 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:54:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:33390->10.242.204.71:8302: i/o timeout 2021/06/05 11:54:56 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:54:56 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/06/05 11:54:56 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 11:54:56 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/05 11:54:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:54:57 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 11:54:57 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:54:57 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/06/05 11:54:57 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-5tc7t 10.242.7.74 2021/06/05 11:54:57 [INFO] consul: member 'dev-consul-7467bf7b46-5tc7t' failed, marking health critical 2021/06/05 11:54:57 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 11:54:57 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/05 11:54:57 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-5tc7t 10.242.7.74 2021/06/05 11:54:57 [INFO] consul: member 'dev-consul-7467bf7b46-5tc7t' joined, marking health alive 2021/06/05 11:54:57 [WARN] raft: Rejecting vote request from 10.242.204.71:8300 since we have a leader: 10.242.7.75:8300 2021/06/05 11:54:57 [INFO] raft: pipelining replication to peer {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300} 2021/06/05 11:55:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:55:20 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/05 11:55:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:39002->10.242.204.71:8301: i/o timeout 2021/06/05 11:55:30 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:55:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:55:32 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured 2021/06/05 11:55:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:55:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:55:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:39300->10.242.204.71:8301: i/o timeout 2021/06/05 11:55:36 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:55:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:55:37 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500167545s 2021/06/05 11:55:37 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 11:55:37 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 11:55:37 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:55:37 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical 2021/06/05 11:55:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:55:39 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 4.9946887s 2021/06/05 11:55:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:55:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:34238->10.242.204.71:8302: i/o timeout 2021/06/05 11:55:41 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:55:42 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 7.494686748s 2021/06/05 11:55:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:55:45 [INFO] raft: aborting pipeline replication to peer {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300} 2021/06/05 11:55:45 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:55:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:55:45 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:51306->10.242.204.71:8300: i/o timeout 2021/06/05 11:55:46 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:55:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:55:47 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.503687865s 2021/06/05 11:55:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:55:50 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 4.989029163s 2021/06/05 11:55:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:55:52 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 7.415832343s 2021/06/05 11:55:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:55:53 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/05 11:55:53 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/06/05 11:55:55 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:52268->10.242.204.71:8300: i/o timeout 2021/06/05 11:55:55 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:55:56 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:52304->10.242.204.71:8300: i/o timeout 2021/06/05 11:55:56 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:55:56 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/05 11:55:56 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:55:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:34508->10.242.204.71:8302: i/o timeout 2021/06/05 11:55:56 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:55:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:55:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:01 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.204.71:8301 2021/06/05 11:56:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:05 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:52450->10.242.204.71:8300: i/o timeout 2021/06/05 11:56:05 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:56:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:06 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:52504->10.242.204.71:8300: i/o timeout 2021/06/05 11:56:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:34698->10.242.204.71:8302: i/o timeout 2021/06/05 11:56:06 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:56:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:07 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:56:07 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/06/05 11:56:07 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 11:56:07 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/05 11:56:07 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/05 11:56:07 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.204.71:52372 2021/06/05 11:56:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:15 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:52640->10.242.204.71:8300: i/o timeout 2021/06/05 11:56:15 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:56:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:17 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:52714->10.242.204.71:8300: i/o timeout 2021/06/05 11:56:18 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:56:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:25 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:52832->10.242.204.71:8300: i/o timeout 2021/06/05 11:56:25 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:56:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:28 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:52918->10.242.204.71:8300: i/o timeout 2021/06/05 11:56:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:29 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:56:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:35 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:53014->10.242.204.71:8300: i/o timeout 2021/06/05 11:56:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:35 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/05 11:56:35 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/06/05 11:56:35 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:56:38 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:56:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:39 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:53120->10.242.204.71:8300: i/o timeout 2021/06/05 11:56:39 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:56:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:41 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.204.71:8301 2021/06/05 11:56:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:45 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:53196->10.242.204.71:8300: i/o timeout 2021/06/05 11:56:45 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:56:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:49 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:53302->10.242.204.71:8300: i/o timeout 2021/06/05 11:56:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:51 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:56:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:55 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:53408->10.242.204.71:8300: i/o timeout 2021/06/05 11:56:55 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:56:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:56:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:01 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:53520->10.242.204.71:8300: i/o timeout 2021/06/05 11:57:02 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:57:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:05 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:53594->10.242.204.71:8300: i/o timeout 2021/06/05 11:57:06 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:57:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:12 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:53740->10.242.204.71:8300: i/o timeout 2021/06/05 11:57:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:14 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:57:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:16 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:53804->10.242.204.71:8300: i/o timeout 2021/06/05 11:57:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:17 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/05 11:57:17 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/06/05 11:57:17 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:57:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:57:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:21 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.204.71:8301 2021/06/05 11:57:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:24 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:53948->10.242.204.71:8300: i/o timeout 2021/06/05 11:57:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:27 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:54022->10.242.204.71:8300: i/o timeout 2021/06/05 11:57:28 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:57:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:30 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:57:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:31 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.204.71:8302 2021/06/05 11:57:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:38 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:54216->10.242.204.71:8300: i/o timeout 2021/06/05 11:57:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:40 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:54254->10.242.204.71:8300: i/o timeout 2021/06/05 11:57:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:44 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:57:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:45 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:57:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:54 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:54494->10.242.204.71:8300: i/o timeout 2021/06/05 11:57:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:55 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:54528->10.242.204.71:8300: i/o timeout 2021/06/05 11:57:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:57:59 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/05 11:57:59 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/06/05 11:58:00 [WARN] raft: Rejecting vote request from 10.242.204.71:8300 since we have a leader: 10.242.7.75:8300 2021/06/05 11:58:00 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 11:58:00 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:58:00 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/06/05 11:58:00 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 11:58:00 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/05 11:58:01 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 11:58:05 [ERR] raft: peer {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300} has newer term, stopping replication 2021/06/05 11:58:05 [INFO] raft: Node at 10.242.7.75:8300 [Follower] entering Follower state (Leader: "") 2021/06/05 11:58:05 [INFO] raft: aborting pipeline replication to peer {Voter d942fbe0-30f7-c6b7-2f54-9c43c9c150dd 10.242.37.70:8300} 2021/06/05 11:58:05 [INFO] consul: cluster leadership lost 2021/06/05 11:58:09 [WARN] raft: Rejecting vote request from 10.242.204.71:8300 since our last index is greater (915, 889) 2021/06/05 11:58:14 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.79.8:39480 2021/06/05 11:58:15 [WARN] raft: Heartbeat timeout from "" reached, starting election 2021/06/05 11:58:15 [INFO] raft: Node at 10.242.7.75:8300 [Candidate] entering Candidate state in term 7 2021/06/05 11:58:15 [INFO] raft: Election won. Tally: 2 2021/06/05 11:58:15 [INFO] raft: Node at 10.242.7.75:8300 [Leader] entering Leader state 2021/06/05 11:58:15 [INFO] raft: Added peer d942fbe0-30f7-c6b7-2f54-9c43c9c150dd, starting replication 2021/06/05 11:58:15 [INFO] raft: Added peer 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, starting replication 2021/06/05 11:58:15 [INFO] consul: cluster leadership acquired 2021/06/05 11:58:15 [INFO] consul: New leader elected: dev-consul-server-0 2021/06/05 11:58:15 [INFO] raft: pipelining replication to peer {Voter d942fbe0-30f7-c6b7-2f54-9c43c9c150dd 10.242.37.70:8300} 2021/06/05 11:58:15 [WARN] raft: AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300} rejected, sending older logs (next: 890) 2021/06/05 11:58:15 [INFO] raft: pipelining replication to peer {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300} 2021/06/05 11:59:00 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:59:08 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 11:59:10 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:59:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:59:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:43350->10.242.204.71:8301: i/o timeout 2021/06/05 11:59:12 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 11:59:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:59:13 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500252562s 2021/06/05 11:59:14 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 11:59:14 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:59:14 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical 2021/06/05 11:59:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:59:15 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:59:16 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:59:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:59:17 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:59:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:59:19 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500160656s 2021/06/05 11:59:20 [INFO] raft: aborting pipeline replication to peer {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300} 2021/06/05 11:59:20 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:59:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:59:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:38354->10.242.204.71:8302: i/o timeout 2021/06/05 11:59:21 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:59:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:59:23 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500220219s 2021/06/05 11:59:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:59:25 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 4.935158445s 2021/06/05 11:59:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:59:27 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:54892->10.242.204.71:8300: i/o timeout 2021/06/05 11:59:28 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 7.343598846s 2021/06/05 11:59:28 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:59:29 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/05 11:59:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:59:30 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:56306->10.242.204.71:8300: i/o timeout 2021/06/05 11:59:30 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:59:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:59:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:38440->10.242.204.71:8302: i/o timeout 2021/06/05 11:59:31 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:59:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:59:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:59:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:59:38 [ERR] raft: Failed to heartbeat to 10.242.204.71:8300: read tcp 10.242.7.75:56464->10.242.204.71:8300: i/o timeout 2021/06/05 11:59:39 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:59:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:59:40 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:56498->10.242.204.71:8300: i/o timeout 2021/06/05 11:59:40 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 11:59:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:59:41 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/05 11:59:41 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/06/05 11:59:44 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:59:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:59:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:38818->10.242.204.71:8302: i/o timeout 2021/06/05 11:59:46 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 11:59:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:59:48 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 11:59:48 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 11:59:48 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/06/05 11:59:48 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 11:59:48 [WARN] raft: Rejecting vote request from 10.242.204.71:8300 since we have a leader: 10.242.7.75:8300 2021/06/05 11:59:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:59:49 [ERR] raft: peer {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300} has newer term, stopping replication 2021/06/05 11:59:49 [INFO] raft: Node at 10.242.7.75:8300 [Follower] entering Follower state (Leader: "") 2021/06/05 11:59:49 [INFO] raft: aborting pipeline replication to peer {Voter d942fbe0-30f7-c6b7-2f54-9c43c9c150dd 10.242.37.70:8300} 2021/06/05 11:59:49 [INFO] consul: cluster leadership lost 2021/06/05 11:59:52 [WARN] consul.coordinate: Batch update failed: node is not the leader 2021/06/05 11:59:55 [WARN] raft: Heartbeat timeout from "" reached, starting election 2021/06/05 11:59:55 [INFO] raft: Node at 10.242.7.75:8300 [Candidate] entering Candidate state in term 8 2021/06/05 11:59:55 [INFO] raft: Election won. Tally: 2 2021/06/05 11:59:55 [INFO] raft: Node at 10.242.7.75:8300 [Leader] entering Leader state 2021/06/05 11:59:55 [INFO] raft: Added peer d942fbe0-30f7-c6b7-2f54-9c43c9c150dd, starting replication 2021/06/05 11:59:55 [INFO] raft: Added peer 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, starting replication 2021/06/05 11:59:55 [INFO] consul: cluster leadership acquired 2021/06/05 11:59:55 [INFO] consul: New leader elected: dev-consul-server-0 2021/06/05 11:59:55 [INFO] raft: pipelining replication to peer {Voter d942fbe0-30f7-c6b7-2f54-9c43c9c150dd 10.242.37.70:8300} 2021/06/05 11:59:56 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured 2021/06/05 11:59:57 [WARN] raft: AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300} rejected, sending older logs (next: 928) 2021/06/05 11:59:58 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:59:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:59:59 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 2.500216379s 2021/06/05 11:59:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:44256->10.242.204.71:8301: i/o timeout 2021/06/05 11:59:59 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:00:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 12:00:01 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 12:00:01 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:00:01 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical 2021/06/05 12:00:02 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 4.94804604s 2021/06/05 12:00:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 12:00:04 [WARN] raft: Failed to contact 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 in 7.41581542s 2021/06/05 12:00:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 12:00:07 [ERR] raft: Failed to AppendEntries to {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300}: read tcp 10.242.7.75:56658->10.242.204.71:8300: i/o timeout 2021/06/05 12:00:07 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 12:00:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 12:00:08 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 12:00:09 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 12:00:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 12:00:10 [WARN] Unable to get address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9, using fallback address 10.242.204.71:8300: Could not find address for server id 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 2021/06/05 12:00:10 [WARN] raft: Rejecting vote request from 10.242.204.71:8300 since we have a leader: 10.242.7.75:8300 2021/06/05 12:00:10 [ERR] raft: peer {Voter 56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9 10.242.204.71:8300} has newer term, stopping replication 2021/06/05 12:00:10 [INFO] raft: Node at 10.242.7.75:8300 [Follower] entering Follower state (Leader: "") 2021/06/05 12:00:10 [INFO] raft: aborting pipeline replication to peer {Voter d942fbe0-30f7-c6b7-2f54-9c43c9c150dd 10.242.37.70:8300} 2021/06/05 12:00:10 [INFO] consul: cluster leadership lost 2021/06/05 12:00:10 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 12:00:10 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:00:16 [INFO] consul: New leader elected: dev-consul-server-2 2021/06/05 12:00:17 [DEBUG] raft-net: 10.242.7.75:8300 accepted connection from: 10.242.37.70:33472 2021/06/05 12:00:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:44590->10.242.204.71:8301: i/o timeout 2021/06/05 12:00:17 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:00:19 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 12:00:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:44636->10.242.204.71:8301: i/o timeout 2021/06/05 12:00:21 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:00:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:44702->10.242.204.71:8301: i/o timeout 2021/06/05 12:00:24 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:00:25 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 12:00:25 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 12:00:25 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:00:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:39578->10.242.204.71:8302: i/o timeout 2021/06/05 12:00:26 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:00:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:44756->10.242.204.71:8301: i/o timeout 2021/06/05 12:00:27 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:00:30 [ERR] memberlist: Push/Pull with dev-consul-server-1 failed: read tcp 10.242.7.75:44634->10.242.204.71:8301: i/o timeout 2021/06/05 12:00:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:39866->10.242.204.71:8302: i/o timeout 2021/06/05 12:00:41 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:00:48 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/05 12:00:48 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.204.71:53546 2021/06/05 12:00:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:40058->10.242.204.71:8302: i/o timeout 2021/06/05 12:00:51 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:00:53 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/05 12:00:56 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/06/05 12:00:56 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 12:00:56 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/05 12:01:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:40132->10.242.204.71:8302: i/o timeout 2021/06/05 12:01:01 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:02:01 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.204.71:8302 2021/06/05 12:02:25 [ERR] memberlist: failed to receive: read tcp 10.242.7.75:8302->10.242.204.71:48688: i/o timeout from=10.242.204.71:48688 2021/06/05 12:02:25 [ERR] memberlist: Failed to send error: write tcp 10.242.7.75:8302->10.242.204.71:48688: i/o timeout from=10.242.204.71:48688 2021/06/05 12:02:25 [ERR] memberlist: failed to receive: read tcp 10.242.7.75:8302->10.242.204.71:48690: i/o timeout from=10.242.204.71:48690 2021/06/05 12:02:25 [ERR] memberlist: Failed to send error: write tcp 10.242.7.75:8302->10.242.204.71:48690: i/o timeout from=10.242.204.71:48690 2021/06/05 12:03:01 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.204.71:8302 2021/06/05 12:04:00 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/05 12:04:00 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 12:04:00 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/05 12:04:00 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 12:04:00 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:05:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:49894->10.242.204.71:8301: i/o timeout 2021/06/05 12:05:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:49908->10.242.204.71:8301: i/o timeout 2021/06/05 12:05:05 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:05:08 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 12:05:08 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:05:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:50006->10.242.204.71:8301: i/o timeout 2021/06/05 12:05:09 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:05:10 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 12:05:10 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:05:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:50058->10.242.204.71:8301: i/o timeout 2021/06/05 12:05:12 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:05:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:50088->10.242.204.71:8301: i/o timeout 2021/06/05 12:05:15 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:05:16 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 12:05:16 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 12:05:16 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:05:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:45262->10.242.204.71:8302: i/o timeout 2021/06/05 12:05:31 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:05:31 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.204.71:8301 2021/06/05 12:05:41 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/05 12:05:41 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.204.71:56556 2021/06/05 12:05:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:45436->10.242.204.71:8302: i/o timeout 2021/06/05 12:05:46 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:05:57 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/06/05 12:05:57 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 12:05:57 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/05 12:06:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:45810->10.242.204.71:8302: i/o timeout 2021/06/05 12:06:01 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:06:12 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/05 12:06:31 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.204.71:8302 2021/06/05 12:06:37 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 12:06:37 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:06:37 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 12:06:37 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/05 12:06:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:51640->10.242.204.71:8301: i/o timeout 2021/06/05 12:06:38 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:06:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:51718->10.242.204.71:8301: i/o timeout 2021/06/05 12:06:42 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:06:45 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 12:06:45 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:06:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:46736->10.242.204.71:8302: i/o timeout 2021/06/05 12:06:51 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:06:55 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 12:06:55 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:06:56 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 12:06:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:52042->10.242.204.71:8301: i/o timeout 2021/06/05 12:06:59 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:07:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:52058->10.242.204.71:8301: i/o timeout 2021/06/05 12:07:02 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:07:03 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 12:07:03 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 12:07:03 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:07:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:47102->10.242.204.71:8302: i/o timeout 2021/06/05 12:07:11 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:07:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:47180->10.242.204.71:8302: i/o timeout 2021/06/05 12:07:21 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:07:28 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 12:07:28 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:07:29 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/05 12:08:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:53972->10.242.204.71:8301: i/o timeout 2021/06/05 12:08:45 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:08:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:54034->10.242.204.71:8301: i/o timeout 2021/06/05 12:08:47 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:08:48 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 12:08:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:54126->10.242.204.71:8301: i/o timeout 2021/06/05 12:08:53 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:08:54 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 12:08:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:54260->10.242.204.71:8301: i/o timeout 2021/06/05 12:08:59 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:09:03 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 12:09:03 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 12:09:03 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:09:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:54320->10.242.204.71:8301: i/o timeout 2021/06/05 12:09:03 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:09:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:49392->10.242.204.71:8302: i/o timeout 2021/06/05 12:09:16 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:09:17 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 12:09:17 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:09:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:54638->10.242.204.71:8301: i/o timeout 2021/06/05 12:09:20 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:09:23 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 12:09:23 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:09:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:54706->10.242.204.71:8301: i/o timeout 2021/06/05 12:09:24 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:09:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:49764->10.242.204.71:8302: i/o timeout 2021/06/05 12:09:36 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:09:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:49950->10.242.204.71:8302: i/o timeout 2021/06/05 12:09:46 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:10:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:50244->10.242.204.71:8302: i/o timeout 2021/06/05 12:10:01 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:10:06 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/06/05 12:10:06 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 12:10:06 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/05 12:10:07 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.204.71:8302 2021/06/05 12:10:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:50320->10.242.204.71:8302: i/o timeout 2021/06/05 12:10:11 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:10:11 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.204.71:8301 2021/06/05 12:10:27 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 12:10:32 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 12:10:32 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:10:33 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 12:10:33 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/05 12:10:33 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/05 12:10:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:56102->10.242.204.71:8301: i/o timeout 2021/06/05 12:10:39 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:10:40 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 12:10:41 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 12:10:41 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:10:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:56118->10.242.204.71:8301: i/o timeout 2021/06/05 12:10:42 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:10:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:50978->10.242.204.71:8302: i/o timeout 2021/06/05 12:10:52 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 12:10:52 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:10:53 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/05 12:10:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:56410->10.242.204.71:8301: i/o timeout 2021/06/05 12:10:56 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:10:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:56488->10.242.204.71:8301: i/o timeout 2021/06/05 12:10:59 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:10:59 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 12:10:59 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:11:12 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 12:11:12 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:11:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:56766->10.242.204.71:8301: i/o timeout 2021/06/05 12:11:15 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:11:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:56826->10.242.204.71:8301: i/o timeout 2021/06/05 12:11:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:56868->10.242.204.71:8301: i/o timeout 2021/06/05 12:11:19 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:11:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:56918->10.242.204.71:8301: i/o timeout 2021/06/05 12:11:22 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:12:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:58508->10.242.204.71:8301: i/o timeout 2021/06/05 12:12:48 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:12:51 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured 2021/06/05 12:12:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:58584->10.242.204.71:8301: i/o timeout 2021/06/05 12:12:53 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:12:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:58636->10.242.204.71:8301: i/o timeout 2021/06/05 12:12:56 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:12:57 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 12:12:57 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 12:12:57 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:13:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:53662->10.242.204.71:8302: i/o timeout 2021/06/05 12:13:06 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:13:11 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 12:13:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:53776->10.242.204.71:8302: i/o timeout 2021/06/05 12:13:16 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:13:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:54036->10.242.204.71:8302: i/o timeout 2021/06/05 12:13:26 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:13:32 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/05 12:13:33 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 12:13:33 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:13:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:59474->10.242.204.71:8301: i/o timeout 2021/06/05 12:13:41 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:13:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:59532->10.242.204.71:8301: i/o timeout 2021/06/05 12:13:44 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:13:45 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 12:13:45 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:13:46 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 12:13:46 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:13:48 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured 2021/06/05 12:13:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:59698->10.242.204.71:8301: i/o timeout 2021/06/05 12:13:52 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:13:53 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 12:13:53 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 12:13:53 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:13:54 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 12:13:54 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:13:55 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured 2021/06/05 12:14:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:59930->10.242.204.71:8301: i/o timeout 2021/06/05 12:14:05 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:14:09 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 12:14:09 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 12:14:09 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:14:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:60028->10.242.204.71:8301: i/o timeout 2021/06/05 12:14:14 [ERR] memberlist: failed to receive: read tcp 10.242.7.75:8301->10.242.204.71:58768: i/o timeout from=10.242.204.71:58768 2021/06/05 12:14:14 [ERR] memberlist: Failed to send error: write tcp 10.242.7.75:8301->10.242.204.71:58768: i/o timeout from=10.242.204.71:58768 2021/06/05 12:14:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:55160->10.242.204.71:8302: i/o timeout 2021/06/05 12:14:26 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:14:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:55344->10.242.204.71:8302: i/o timeout 2021/06/05 12:14:36 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:14:41 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 12:14:41 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 12:14:41 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:15:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:32802->10.242.204.71:8301: i/o timeout 2021/06/05 12:15:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:32978->10.242.204.71:8301: i/o timeout 2021/06/05 12:15:14 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:15:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:33058->10.242.204.71:8301: i/o timeout 2021/06/05 12:15:17 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:15:17 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 12:15:17 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:15:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:56278->10.242.204.71:8302: i/o timeout 2021/06/05 12:15:26 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:15:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:56456->10.242.204.71:8302: i/o timeout 2021/06/05 12:15:36 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:15:43 [WARN] raft: Rejecting vote request from 10.242.204.71:8300 since we have a leader: 10.242.37.70:8300 2021/06/05 12:15:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:56646->10.242.204.71:8302: i/o timeout 2021/06/05 12:15:46 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:15:48 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 12:15:48 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:15:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:33652->10.242.204.71:8301: i/o timeout 2021/06/05 12:15:49 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:15:53 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 12:15:53 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 12:15:53 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:15:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:33728->10.242.204.71:8301: i/o timeout 2021/06/05 12:15:54 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:15:56 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured 2021/06/05 12:16:07 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 12:16:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:57128->10.242.204.71:8302: i/o timeout 2021/06/05 12:16:11 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:16:14 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 12:16:14 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:16:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:34116->10.242.204.71:8301: i/o timeout 2021/06/05 12:16:15 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:16:16 [WARN] raft: Rejecting vote request from 10.242.204.71:8300 since we have a leader: 10.242.37.70:8300 2021/06/05 12:16:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:34548->10.242.204.71:8301: i/o timeout 2021/06/05 12:16:38 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:16:41 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 12:16:41 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 12:16:41 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:16:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:57682->10.242.204.71:8302: i/o timeout 2021/06/05 12:16:41 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:16:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:57874->10.242.204.71:8302: i/o timeout 2021/06/05 12:16:51 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:17:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:57940->10.242.204.71:8302: i/o timeout 2021/06/05 12:17:01 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:17:09 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 12:17:09 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:17:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:35142->10.242.204.71:8301: i/o timeout 2021/06/05 12:17:10 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:17:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:58236->10.242.204.71:8302: i/o timeout 2021/06/05 12:17:11 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:17:11 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/06/05 12:17:11 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 12:17:11 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/05 12:17:12 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 12:17:12 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 12:17:12 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/05 12:20:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:39020->10.242.204.71:8301: i/o timeout 2021/06/05 12:20:43 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:20:46 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 12:20:46 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 12:20:46 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:20:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:39040->10.242.204.71:8301: i/o timeout 2021/06/05 12:20:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:34094->10.242.204.71:8302: i/o timeout 2021/06/05 12:20:56 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:21:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:34290->10.242.204.71:8302: i/o timeout 2021/06/05 12:21:06 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:21:17 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/06/05 12:21:17 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 12:21:17 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/05 12:21:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:34474->10.242.204.71:8302: i/o timeout 2021/06/05 12:21:21 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:21:21 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.204.71:8301 2021/06/05 12:21:32 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.204.71:8302 2021/06/05 12:22:01 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.204.71:8301 2021/06/05 12:22:39 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 12:22:39 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/05 12:22:39 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 12:22:39 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:22:39 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/05 12:22:39 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 12:23:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:41604->10.242.204.71:8301: i/o timeout 2021/06/05 12:23:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:41710->10.242.204.71:8301: i/o timeout 2021/06/05 12:23:08 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:23:09 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 12:23:09 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 12:23:09 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:23:11 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.204.71:8301 2021/06/05 12:23:16 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 12:23:16 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:23:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:41880->10.242.204.71:8301: i/o timeout 2021/06/05 12:23:17 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:23:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:41928->10.242.204.71:8301: i/o timeout 2021/06/05 12:23:20 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:23:21 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 12:23:21 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 12:23:21 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:23:26 [ERR] memberlist: failed to receive: read tcp 10.242.7.75:8301->10.242.204.71:37550: i/o timeout from=10.242.204.71:37550 2021/06/05 12:23:26 [ERR] memberlist: Failed to send error: write tcp 10.242.7.75:8301->10.242.204.71:37550: i/o timeout from=10.242.204.71:37550 2021/06/05 12:23:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:36874->10.242.204.71:8302: i/o timeout 2021/06/05 12:23:26 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:23:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:36988->10.242.204.71:8302: i/o timeout 2021/06/05 12:23:36 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:23:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:37236->10.242.204.71:8302: i/o timeout 2021/06/05 12:23:46 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:23:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:37422->10.242.204.71:8302: i/o timeout 2021/06/05 12:23:56 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:23:56 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/06/05 12:23:56 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 12:23:56 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/05 12:24:08 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 12:24:08 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:24:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:42836->10.242.204.71:8301: i/o timeout 2021/06/05 12:24:09 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:24:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:42896->10.242.204.71:8301: i/o timeout 2021/06/05 12:24:12 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:24:13 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/05 12:24:13 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 12:24:13 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:24:27 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 12:24:27 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:24:27 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/05 12:24:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:43220->10.242.204.71:8301: i/o timeout 2021/06/05 12:24:29 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:24:32 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.204.71:8302 2021/06/05 12:24:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:43250->10.242.204.71:8301: i/o timeout 2021/06/05 12:24:32 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/05 12:24:32 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.204.71 2021/06/05 12:24:32 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:24:51 [ERR] memberlist: failed to receive: read tcp 10.242.7.75:8301->10.242.204.71:38872: i/o timeout from=10.242.204.71:38872 2021/06/05 12:24:51 [ERR] memberlist: Failed to send error: write tcp 10.242.7.75:8301->10.242.204.71:38872: i/o timeout from=10.242.204.71:38872 2021/06/05 12:25:13 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 12:25:13 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/05 12:25:15 [WARN] memberlist: Refuting a dead message (from: dev-consul-server-1.dc1) 2021/06/05 12:25:16 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.204.71:8301 2021/06/05 12:25:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:39084->10.242.204.71:8302: i/o timeout 2021/06/05 12:25:26 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:25:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:39264->10.242.204.71:8302: i/o timeout 2021/06/05 12:25:36 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:25:46 [ERR] memberlist: Push/Pull with dev-consul-server-1.dc1 failed: read tcp 10.242.7.75:38944->10.242.204.71:8302: i/o timeout 2021/06/05 12:25:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:39570->10.242.204.71:8302: i/o timeout 2021/06/05 12:25:51 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:25:56 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.204.71:8301 2021/06/05 12:25:56 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/06/05 12:25:56 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 12:25:56 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/05 12:26:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.7.75:39634->10.242.204.71:8302: i/o timeout 2021/06/05 12:26:01 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/05 12:27:06 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.204.71:8301 2021/06/05 12:27:11 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 12:27:11 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 12:27:11 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 12:27:11 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 12:27:11 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"