By type
+ CONSUL_BIND= + [ -n ] + CONSUL_CLIENT= + [ -n ] + CONSUL_DATA_DIR=/consul/data + CONSUL_CONFIG_DIR=/consul/config + [ -n ] + [ a = - ] + [ agent = agent ] + shift + set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui + [ consul = consul ] + stat -c %u /consul/data + id -u consul + [ 100 != 100 ] + stat -c %u /consul/config + id -u consul + [ 100 != 100 ] + [ ! -z ] + exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui bootstrap_expect > 0: expecting 3 servers ==> Starting Consul agent... ==> Consul agent running! Version: 'v1.0.6' Node ID: '56de5cf5-271f-0e8e-fc20-cd7d40f1c7f9' Node name: 'dev-consul-server-1' Datacenter: 'dc1' (Segment: '
') Server: true (Bootstrap: false) Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600) Cluster Addr: 10.242.204.71 (LAN: 8301, WAN: 8302) Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false ==> Log data will now stream in as it occurs: 2021/06/05 10:40:23 [INFO] raft: Initial configuration (index=0): [] 2021/06/05 10:40:23 [INFO] raft: Node at 10.242.204.71:8300 [Follower] entering Follower state (Leader: "") 2021/06/05 10:40:23 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.204.71 2021/06/05 10:40:23 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.204.71 2021/06/05 10:40:23 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/05 10:40:23 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp) 2021/06/05 10:40:23 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.204.71:8300) (DC: dc1) 2021/06/05 10:40:23 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp) 2021/06/05 10:40:23 [INFO] agent: Started HTTP server on [::]:8500 (tcp) 2021/06/05 10:40:23 [INFO] agent: started state syncer 2021/06/05 10:40:23 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer 2021/06/05 10:40:23 [INFO] agent: Joining LAN cluster... 2021/06/05 10:40:23 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server] 2021/06/05 10:40:23 [WARN] memberlist: Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host 2021/06/05 10:40:23 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host 2021/06/05 10:40:23 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host 2021/06/05 10:40:23 [INFO] agent: (LAN) joined: 0 Err: 3 error(s) occurred: * Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host * Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host * Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host 2021/06/05 10:40:23 [WARN] agent: Join LAN failed: , retrying in 30s 2021/06/05 10:40:30 [ERR] agent: failed to sync remote state: No cluster leader 2021/06/05 10:40:32 [WARN] raft: no known peers, aborting election ==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=45b41d93-3f82-ac1c-ecb2-deb42c2b1cff&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 2021/06/05 10:40:52 [ERR] agent: Coordinate update error: No cluster leader 2021/06/05 10:40:53 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server] 2021/06/05 10:40:53 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-5tc7t 10.242.7.74 2021/06/05 10:40:53 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.7.75 2021/06/05 10:40:53 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.37.70 2021/06/05 10:40:53 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.7.75:8300) (DC: dc1) 2021/06/05 10:40:53 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.242.204.71:8300,10.242.7.75:8300,10.242.37.70:8300 2021/06/05 10:40:53 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.37.70:8300) (DC: dc1) 2021/06/05 10:40:53 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host 2021/06/05 10:40:53 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.37.70 2021/06/05 10:40:53 [INFO] agent: (LAN) joined: 2 Err: 2021/06/05 10:40:53 [INFO] agent: Join LAN completed. Synced with 2 initial agents 2021/06/05 10:40:53 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.7.75 2021/06/05 10:40:53 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan" 2021/06/05 10:40:53 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan" 2021/06/05 10:40:54 [WARN] raft: Heartbeat timeout from "" reached, starting election 2021/06/05 10:40:54 [INFO] raft: Node at 10.242.204.71:8300 [Candidate] entering Candidate state in term 2 2021/06/05 10:40:54 [INFO] raft: Election won. Tally: 2 2021/06/05 10:40:54 [INFO] raft: Node at 10.242.204.71:8300 [Leader] entering Leader state 2021/06/05 10:40:54 [INFO] raft: Added peer 5fa9f731-8cb6-e040-8d5f-1f0ab18b143a, starting replication 2021/06/05 10:40:54 [INFO] raft: Added peer d942fbe0-30f7-c6b7-2f54-9c43c9c150dd, starting replication 2021/06/05 10:40:54 [INFO] consul: cluster leadership acquired 2021/06/05 10:40:54 [INFO] consul: New leader elected: dev-consul-server-1 2021/06/05 10:40:54 [WARN] raft: AppendEntries to {Voter d942fbe0-30f7-c6b7-2f54-9c43c9c150dd 10.242.37.70:8300} rejected, sending older logs (next: 1) 2021/06/05 10:40:54 [INFO] raft: pipelining replication to peer {Voter 5fa9f731-8cb6-e040-8d5f-1f0ab18b143a 10.242.7.75:8300} 2021/06/05 10:40:54 [INFO] raft: pipelining replication to peer {Voter d942fbe0-30f7-c6b7-2f54-9c43c9c150dd 10.242.37.70:8300} 2021/06/05 10:40:54 [INFO] consul: member 'dev-consul-7467bf7b46-5tc7t' joined, marking health alive 2021/06/05 10:40:54 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive 2021/06/05 10:40:54 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive 2021/06/05 10:40:54 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/06/05 10:40:56 [INFO] agent: Synced node info 2021/06/05 11:45:24 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:38486->10.242.7.75:8301: i/o timeout 2021/06/05 11:45:24 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/05 11:45:26 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2) 2021/06/05 11:45:29 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:59116->10.242.7.74:8301: i/o timeout 2021/06/05 11:45:29 [INFO] memberlist: Suspect dev-consul-7467bf7b46-5tc7t has failed, no acks received 2021/06/05 11:46:41 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/06/05 11:46:41 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/05 11:46:41 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/05 11:46:43 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 11:46:43 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 11:46:43 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:41774 2021/06/05 11:46:43 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:41776 2021/06/05 11:46:43 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:41686 2021/06/05 11:46:43 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:41808 2021/06/05 11:46:43 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:42016 2021/06/05 11:46:43 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:41168 2021/06/05 11:46:43 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:41972 2021/06/05 11:46:43 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: stream closed 2021/06/05 11:46:43 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 11:46:43 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 11:46:43 [ERR] raft: peer {Voter 5fa9f731-8cb6-e040-8d5f-1f0ab18b143a 10.242.7.75:8300} has newer term, stopping replication 2021/06/05 11:46:43 [ERR] yamux: Failed to write header: write tcp 10.242.204.71:8300->10.242.37.70:54384: use of closed network connection 2021/06/05 11:46:43 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: stream closed 2021/06/05 11:46:43 [ERR] raft: peer {Voter d942fbe0-30f7-c6b7-2f54-9c43c9c150dd 10.242.37.70:8300} has newer term, stopping replication 2021/06/05 11:46:43 [INFO] raft: aborting pipeline replication to peer {Voter d942fbe0-30f7-c6b7-2f54-9c43c9c150dd 10.242.37.70:8300} 2021/06/05 11:46:43 [INFO] raft: aborting pipeline replication to peer {Voter 5fa9f731-8cb6-e040-8d5f-1f0ab18b143a 10.242.7.75:8300} 2021/06/05 11:46:43 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:51138->10.242.37.70:8301: i/o timeout 2021/06/05 11:46:43 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/06/05 11:46:44 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/05 11:46:44 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 11:46:44 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:38108->10.242.7.75:8302: i/o timeout 2021/06/05 11:46:44 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/06/05 11:46:44 [INFO] consul: New leader elected: dev-consul-server-0 2021/06/05 11:46:44 [INFO] raft: Node at 10.242.204.71:8300 [Follower] entering Follower state (Leader: "") 2021/06/05 11:46:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/05 11:46:44 [ERR] raft: peer {Voter d942fbe0-30f7-c6b7-2f54-9c43c9c150dd 10.242.37.70:8300} has newer term, stopping replication 2021/06/05 11:46:44 [WARN] consul.coordinate: Batch update failed: leadership lost while committing log 2021/06/05 11:46:44 [ERR] raft: peer {Voter 5fa9f731-8cb6-e040-8d5f-1f0ab18b143a 10.242.7.75:8300} has newer term, stopping replication 2021/06/05 11:46:44 [WARN] raft: Rejecting vote request from 10.242.37.70:8300 since our last index is greater (773, 772) 2021/06/05 11:46:48 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:42162 2021/06/05 11:47:35 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:42350 2021/06/05 11:47:35 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:42988 2021/06/05 11:47:35 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:42352 2021/06/05 11:47:35 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:42538 2021/06/05 11:47:35 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:42566 2021/06/05 11:47:35 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:42732 2021/06/05 11:47:35 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:42764 2021/06/05 11:47:35 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:42934 2021/06/05 11:47:35 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 11:47:35 [WARN] yamux: failed to send ping reply: session shutdown 2021/06/05 11:47:35 [WARN] yamux: failed to send ping reply: session shutdown 2021/06/05 11:47:36 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 11:47:36 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:43126 2021/06/05 11:47:35 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/06/05 11:47:37 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/06/05 11:47:36 [ERR] http: Request PUT /v1/txn, error: rpc error making call: stream closed from=10.242.79.8:46146 2021/06/05 11:47:37 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 11:47:46 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (0 peer confirmations) 2021/06/05 11:47:53 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 11:47:53 [ERR] memberlist: Failed to send ack: write tcp 10.242.204.71:8301->10.242.7.75:57626: i/o timeout from=10.242.7.75:57626 2021/06/05 11:47:53 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:40106->10.242.7.75:8301: i/o timeout 2021/06/05 11:47:53 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:43336 2021/06/05 11:47:53 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/05 11:47:53 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1) 2021/06/05 11:47:53 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:38706->10.242.7.75:8302: i/o timeout 2021/06/05 11:47:53 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/06/05 11:47:53 [WARN] raft: Clearing log suffix from 773 to 773 2021/06/05 11:47:53 [WARN] raft: Rejecting vote request from 10.242.7.75:8300 since our last term is greater (4, 2) 2021/06/05 11:47:53 [WARN] consul.coordinate: Batch update failed: node is not the leader 2021/06/05 11:47:53 [INFO] consul: cluster leadership lost 2021/06/05 11:48:17 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2) 2021/06/05 11:48:17 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:52326->10.242.37.70:8301: i/o timeout 2021/06/05 11:48:17 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/06/05 11:48:39 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:44184 2021/06/05 11:48:40 [ERR] agent: Coordinate update error: rpc error making call: stream closed 2021/06/05 11:48:40 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 11:48:40 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:44208 2021/06/05 11:48:40 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:33316->10.242.7.74:8301: i/o timeout 2021/06/05 11:48:40 [INFO] memberlist: Suspect dev-consul-7467bf7b46-5tc7t has failed, no acks received 2021/06/05 11:48:40 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2) 2021/06/05 11:48:40 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1) 2021/06/05 11:48:42 [ERR] raft-net: Failed to flush response: write tcp 10.242.204.71:8300->10.242.7.75:43126: write: broken pipe 2021/06/05 11:48:42 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:44376 2021/06/05 11:48:42 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0) 2021/06/05 11:48:50 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0) 2021/06/05 11:49:06 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:41406->10.242.7.75:8301: i/o timeout 2021/06/05 11:49:06 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/05 11:49:06 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0) 2021/06/05 11:49:08 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2) 2021/06/05 11:49:26 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:45206 2021/06/05 11:49:33 [WARN] raft: Heartbeat timeout from "10.242.7.75:8300" reached, starting election 2021/06/05 11:49:34 [INFO] raft: Node at 10.242.204.71:8300 [Candidate] entering Candidate state in term 5 2021/06/05 11:49:34 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:45168 2021/06/05 11:49:34 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 11:49:34 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1) 2021/06/05 11:49:34 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:34110->10.242.7.74:8301: i/o timeout 2021/06/05 11:49:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-5tc7t has failed, no acks received 2021/06/05 11:49:34 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:44636->10.242.37.70:8302: i/o timeout 2021/06/05 11:49:34 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received 2021/06/05 11:49:35 [INFO] raft: Node at 10.242.204.71:8300 [Follower] entering Follower state (Leader: "10.242.7.75:8300") 2021/06/05 11:49:35 [ERR] raft-net: Failed to flush response: write tcp 10.242.204.71:8300->10.242.7.75:44208: write: broken pipe 2021/06/05 11:50:09 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2) 2021/06/05 11:50:26 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:54246->10.242.37.70:8301: i/o timeout 2021/06/05 11:50:26 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/06/05 11:50:30 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2) 2021/06/05 11:50:56 [WARN] raft: Heartbeat timeout from "10.242.7.75:8300" reached, starting election 2021/06/05 11:50:57 [WARN] yamux: failed to send ping reply: session shutdown 2021/06/05 11:50:57 [INFO] raft: Node at 10.242.204.71:8300 [Candidate] entering Candidate state in term 5 2021/06/05 11:51:05 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 11:51:07 [WARN] yamux: failed to send ping reply: session shutdown 2021/06/05 11:51:13 [ERR] memberlist: Failed to send ack: write tcp 10.242.204.71:8301->10.242.7.75:33622: i/o timeout from=10.242.7.75:33622 2021/06/05 11:51:13 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:46802 2021/06/05 11:51:13 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:46622 2021/06/05 11:51:13 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:47020 2021/06/05 11:51:13 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:46606 2021/06/05 11:51:13 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:46820 2021/06/05 11:51:13 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:46996 2021/06/05 11:51:13 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:47180 2021/06/05 11:51:13 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/05 11:51:13 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:54748->10.242.37.70:8301: i/o timeout 2021/06/05 11:51:13 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/06/05 11:51:13 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:41716->10.242.7.75:8302: i/o timeout 2021/06/05 11:51:13 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/06/05 11:51:15 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0) 2021/06/05 11:51:15 [ERR] raft-net: Failed to flush response: write tcp 10.242.204.71:8300->10.242.7.75:45168: write: broken pipe 2021/06/05 11:51:15 [INFO] raft: Node at 10.242.204.71:8300 [Follower] entering Follower state (Leader: "10.242.7.75:8300") 2021/06/05 11:51:53 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:43800->10.242.7.75:8301: i/o timeout 2021/06/05 11:51:53 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/05 11:52:05 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2) 2021/06/05 11:52:05 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:43986->10.242.7.75:8301: i/o timeout 2021/06/05 11:52:05 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/05 11:52:24 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-5tc7t) 2021/06/05 11:52:25 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:44262->10.242.7.75:8301: i/o timeout 2021/06/05 11:52:25 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/05 11:52:26 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:47086->10.242.37.70:8302: i/o timeout 2021/06/05 11:52:26 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received 2021/06/05 11:52:31 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:44298->10.242.7.75:8301: i/o timeout 2021/06/05 11:52:31 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/05 11:52:37 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1) 2021/06/05 11:52:39 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:36738->10.242.7.74:8301: i/o timeout 2021/06/05 11:52:39 [INFO] memberlist: Suspect dev-consul-7467bf7b46-5tc7t has failed, no acks received 2021/06/05 11:52:39 [ERR] raft-net: Failed to flush response: write tcp 10.242.204.71:8300->10.242.7.75:47180: write: broken pipe 2021/06/05 11:52:49 [WARN] raft: Heartbeat timeout from "10.242.7.75:8300" reached, starting election 2021/06/05 11:52:49 [INFO] raft: Node at 10.242.204.71:8300 [Candidate] entering Candidate state in term 5 2021/06/05 11:52:49 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:48990 2021/06/05 11:52:49 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:48794 2021/06/05 11:52:49 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 11:52:50 [INFO] raft: Node at 10.242.204.71:8300 [Follower] entering Follower state (Leader: "10.242.7.75:8300") 2021/06/05 11:53:26 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:49596 2021/06/05 11:53:30 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0) 2021/06/05 11:53:39 [ERR] raft-net: Failed to flush response: write tcp 10.242.204.71:8300->10.242.7.75:48990: write: broken pipe 2021/06/05 11:53:41 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/05 11:53:42 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:49820 2021/06/05 11:53:43 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:43808->10.242.7.75:8302: i/o timeout 2021/06/05 11:53:43 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/06/05 11:53:45 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:49978 2021/06/05 11:53:45 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:49782 2021/06/05 11:53:45 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:50046 2021/06/05 11:53:45 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:45194->10.242.7.75:8301: i/o timeout 2021/06/05 11:53:45 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/05 11:53:58 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:50172 2021/06/05 11:54:03 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:50364 2021/06/05 11:54:02 [ERR] memberlist: Failed to send ack: write tcp 10.242.204.71:8301->10.242.7.74:47824: i/o timeout from=10.242.7.74:47824 2021/06/05 11:54:03 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:50278 2021/06/05 11:54:02 [WARN] yamux: failed to send ping reply: session shutdown 2021/06/05 11:54:03 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:48454->10.242.37.70:8302: i/o timeout 2021/06/05 11:54:03 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received 2021/06/05 11:54:05 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-5tc7t) 2021/06/05 11:54:10 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:45734->10.242.7.75:8301: i/o timeout 2021/06/05 11:54:10 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/05 11:54:15 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1) 2021/06/05 11:54:17 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:38150->10.242.7.74:8301: i/o timeout 2021/06/05 11:54:17 [INFO] memberlist: Suspect dev-consul-7467bf7b46-5tc7t has failed, no acks received 2021/06/05 11:54:18 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 11:54:54 [ERR] yamux: Failed to write header: write tcp 10.242.204.71:8300->10.242.7.75:50332: use of closed network connection 2021/06/05 11:54:54 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 11:54:54 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 11:54:54 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 11:54:57 [ERR] yamux: Failed to write header: write tcp 10.242.204.71:52026->10.242.7.75:8300: use of closed network connection 2021/06/05 11:54:57 [WARN] raft: Heartbeat timeout from "10.242.7.75:8300" reached, starting election 2021/06/05 11:54:57 [ERR] yamux: Failed to write header: write tcp 10.242.204.71:52064->10.242.37.70:8300: use of closed network connection 2021/06/05 11:54:57 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:50880 2021/06/05 11:54:57 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:51306 2021/06/05 11:54:57 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:50900 2021/06/05 11:54:57 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:51058 2021/06/05 11:54:57 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:51100 2021/06/05 11:54:57 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:51248 2021/06/05 11:54:57 [INFO] raft: Node at 10.242.204.71:8300 [Candidate] entering Candidate state in term 5 2021/06/05 11:54:57 [INFO] memberlist: Marking dev-consul-7467bf7b46-5tc7t as failed, suspect timeout reached (0 peer confirmations) 2021/06/05 11:54:57 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:57694->10.242.37.70:8301: i/o timeout 2021/06/05 11:54:57 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/06/05 11:54:57 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-5tc7t 10.242.7.74 2021/06/05 11:54:57 [ERR] memberlist: Failed to send ack: write tcp 10.242.204.71:8301->10.242.37.70:48384: i/o timeout from=10.242.37.70:48384 2021/06/05 11:54:57 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1) 2021/06/05 11:54:57 [INFO] raft: Node at 10.242.204.71:8300 [Follower] entering Follower state (Leader: "10.242.7.75:8300") 2021/06/05 11:54:57 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0) 2021/06/05 11:54:57 [ERR] raft-net: Failed to flush response: write tcp 10.242.204.71:8300->10.242.7.75:50364: write: broken pipe 2021/06/05 11:54:57 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-5tc7t 10.242.7.74 2021/06/05 11:55:40 [WARN] raft: Heartbeat timeout from "10.242.7.75:8300" reached, starting election 2021/06/05 11:58:00 [ERR] agent: Coordinate update error: rpc error making call: stream closed 2021/06/05 11:57:59 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 11:58:00 [INFO] raft: Node at 10.242.204.71:8300 [Candidate] entering Candidate state in term 5 2021/06/05 11:58:00 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 11:58:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:52832 2021/06/05 11:58:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:52504 2021/06/05 11:58:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:52640 2021/06/05 11:58:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:52714 2021/06/05 11:58:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:53196 2021/06/05 11:58:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:52918 2021/06/05 11:58:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:53014 2021/06/05 11:58:00 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:50054->10.242.37.70:8302: i/o timeout 2021/06/05 11:58:00 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received 2021/06/05 11:58:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:53520 2021/06/05 11:58:00 [WARN] yamux: failed to send ping reply: session shutdown 2021/06/05 11:58:00 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 11:58:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:52304 2021/06/05 11:58:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:52450 2021/06/05 11:58:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:53120 2021/06/05 11:58:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:54528 2021/06/05 11:58:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:53594 2021/06/05 11:58:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:53302 2021/06/05 11:58:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:53740 2021/06/05 11:58:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:52268 2021/06/05 11:58:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:53408 2021/06/05 11:58:00 [ERR] memberlist: Failed to send ack: write tcp 10.242.204.71:8301->10.242.7.74:49988: i/o timeout from=10.242.7.74:49988 2021/06/05 11:58:00 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/05 11:58:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:53804 2021/06/05 11:58:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:53948 2021/06/05 11:58:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:54022 2021/06/05 11:58:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:54216 2021/06/05 11:58:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:54254 2021/06/05 11:58:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:54494 2021/06/05 11:58:00 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:47306->10.242.7.75:8301: i/o timeout 2021/06/05 11:58:00 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/05 11:58:01 [ERR] raft-net: Failed to flush response: write tcp 10.242.204.71:8300->10.242.7.75:51248: write: broken pipe 2021/06/05 11:58:04 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:54892 2021/06/05 11:58:07 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.79.8:51100 2021/06/05 11:58:07 [ERR] agent: failed to sync remote state: No cluster leader 2021/06/05 11:58:07 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.79.8:50826 2021/06/05 11:58:09 [WARN] raft: Election timeout reached, restarting election 2021/06/05 11:58:09 [INFO] raft: Node at 10.242.204.71:8300 [Candidate] entering Candidate state in term 6 2021/06/05 11:58:12 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:53154 2021/06/05 11:58:15 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:55078 2021/06/05 11:58:15 [INFO] raft: Node at 10.242.204.71:8300 [Follower] entering Follower state (Leader: "") 2021/06/05 11:58:15 [WARN] raft: Failed to get previous log: 915 log not found (last: 889) 2021/06/05 11:58:15 [INFO] consul: New leader elected: dev-consul-server-0 2021/06/05 11:59:07 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2) 2021/06/05 11:59:08 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:48612->10.242.7.75:8301: i/o timeout 2021/06/05 11:59:08 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/05 11:59:23 [WARN] raft: Heartbeat timeout from "10.242.7.75:8300" reached, starting election 2021/06/05 11:59:27 [ERR] memberlist: Failed to send ack: write tcp 10.242.204.71:8301->10.242.7.74:54138: i/o timeout from=10.242.7.74:54138 2021/06/05 11:59:29 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:47486->10.242.7.75:8302: i/o timeout 2021/06/05 11:59:29 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/06/05 11:59:27 [ERR] memberlist: Failed to send ack: write tcp 10.242.204.71:8301->10.242.7.74:54124: i/o timeout from=10.242.7.74:54124 2021/06/05 11:59:29 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:41180->10.242.7.74:8301: i/o timeout 2021/06/05 11:59:29 [INFO] memberlist: Suspect dev-consul-7467bf7b46-5tc7t has failed, no acks received 2021/06/05 11:59:29 [INFO] raft: Node at 10.242.204.71:8300 [Follower] entering Candidate state in term 8 2021/06/05 11:59:43 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 11:59:44 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2) 2021/06/05 11:59:47 [ERR] memberlist: Failed to send ack: write tcp 10.242.204.71:8301->10.242.37.70:53414: i/o timeout from=10.242.37.70:53414 2021/06/05 11:59:48 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:56464 2021/06/05 11:59:48 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:56306 2021/06/05 11:59:48 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:56498 2021/06/05 11:59:48 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:56658 2021/06/05 11:59:48 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:56676 2021/06/05 11:59:48 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1) 2021/06/05 11:59:48 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:52046->10.242.37.70:8302: i/o timeout 2021/06/05 11:59:48 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received 2021/06/05 11:59:48 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:49222->10.242.7.75:8301: i/o timeout 2021/06/05 11:59:48 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/05 11:59:49 [INFO] raft: Node at 10.242.204.71:8300 [Follower] entering Follower state (Leader: "10.242.7.75:8300") 2021/06/05 11:59:49 [ERR] raft-net: Failed to flush response: write tcp 10.242.204.71:8300->10.242.7.75:55078: write: broken pipe 2021/06/05 11:59:55 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:60956->10.242.37.70:8301: i/o timeout 2021/06/05 11:59:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/06/05 11:59:56 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.7.75:56978 2021/06/05 11:59:57 [INFO] raft: Duplicate RequestVote for same term: 8 2021/06/05 11:59:57 [WARN] raft: Failed to get previous log: 935 log not found (last: 927) 2021/06/05 12:00:07 [WARN] raft: Heartbeat timeout from "10.242.7.75:8300" reached, starting election 2021/06/05 12:00:07 [INFO] consul: New leader elected: dev-consul-server-0 2021/06/05 12:00:10 [INFO] raft: Node at 10.242.204.71:8300 [Follower] entering Candidate state in term 9 2021/06/05 12:00:08 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/05 12:00:10 [INFO] raft: Node at 10.242.204.71:8300 [Follower] entering Follower state (Leader: "10.242.7.75:8300") 2021/06/05 12:00:10 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:41802->10.242.7.74:8301: i/o timeout 2021/06/05 12:00:10 [INFO] memberlist: Suspect dev-consul-7467bf7b46-5tc7t has failed, no acks received 2021/06/05 12:00:10 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-5tc7t) 2021/06/05 12:00:16 [WARN] raft: Rejecting vote request from 10.242.37.70:8300 since we have a leader: 10.242.7.75:8300 2021/06/05 12:00:18 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:55296 2021/06/05 12:00:19 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 12:00:19 [WARN] raft: Failed to get previous log: 940 log not found (last: 927) 2021/06/05 12:00:19 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:49666->10.242.7.75:8301: i/o timeout 2021/06/05 12:00:19 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/05 12:00:19 [INFO] consul: New leader elected: dev-consul-server-2 2021/06/05 12:00:58 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-5tc7t but other probes failed, network may be misconfigured 2021/06/05 12:01:55 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 12:01:55 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 12:01:55 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 12:01:55 [WARN] yamux: failed to send ping reply: session shutdown 2021/06/05 12:01:55 [ERR] yamux: Failed to write header: write tcp 10.242.204.71:8300->10.242.7.75:56728: use of closed network connection 2021/06/05 12:04:00 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1) 2021/06/05 12:04:00 [WARN] yamux: failed to send ping reply: session shutdown 2021/06/05 12:04:00 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (0 peer confirmations) 2021/06/05 12:04:00 [ERR] memberlist: Failed to send ack: write tcp 10.242.204.71:8301->10.242.7.74:55448: i/o timeout from=10.242.7.74:55448 2021/06/05 12:04:00 [ERR] memberlist: Failed to send ack: write tcp 10.242.204.71:8301->10.242.37.70:54656: i/o timeout from=10.242.37.70:54656 2021/06/05 12:04:00 [ERR] memberlist: Failed to send ack: write tcp 10.242.204.71:8301->10.242.7.75:44636: i/o timeout from=10.242.7.75:44636 2021/06/05 12:04:00 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:48690->10.242.7.75:8302: i/o timeout 2021/06/05 12:04:00 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/06/05 12:04:00 [ERR] memberlist: Failed to send ack: write tcp 10.242.204.71:8301->10.242.7.75:44756: i/o timeout from=10.242.7.75:44756 2021/06/05 12:04:00 [ERR] memberlist: failed to receive: read tcp 10.242.204.71:8302->10.242.37.70:59674: i/o timeout from=10.242.37.70:59674 2021/06/05 12:04:00 [ERR] memberlist: Failed to send error: write tcp 10.242.204.71:8302->10.242.37.70:59674: i/o timeout from=10.242.37.70:59674 2021/06/05 12:04:00 [WARN] yamux: failed to send ping reply: session shutdown 2021/06/05 12:04:00 [ERR] memberlist: Push/Pull with dev-consul-server-0.dc1 failed: EOF 2021/06/05 12:04:00 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.7.75 2021/06/05 12:04:00 [ERR] memberlist: Failed to push local state: write tcp 10.242.204.71:8301->10.242.37.70:55466: i/o timeout from=10.242.37.70:55466 2021/06/05 12:04:00 [ERR] memberlist: failed to receive: read tcp 10.242.204.71:8302->10.242.37.70:59942: i/o timeout from=10.242.37.70:59942 2021/06/05 12:04:00 [ERR] memberlist: Failed to send error: write tcp 10.242.204.71:8302->10.242.37.70:59942: i/o timeout from=10.242.37.70:59942 2021/06/05 12:04:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:56868 2021/06/05 12:04:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:55520 2021/06/05 12:04:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:55900 2021/06/05 12:04:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:56000 2021/06/05 12:04:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:55552 2021/06/05 12:04:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:55680 2021/06/05 12:04:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:55730 2021/06/05 12:04:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:56060 2021/06/05 12:04:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:56158 2021/06/05 12:04:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:55842 2021/06/05 12:04:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:56666 2021/06/05 12:04:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:56798 2021/06/05 12:04:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:56244 2021/06/05 12:04:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:56308 2021/06/05 12:04:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:56418 2021/06/05 12:04:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:56484 2021/06/05 12:04:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:56606 2021/06/05 12:04:00 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.7.75:8300) (DC: dc1) 2021/06/05 12:04:01 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:59228 2021/06/05 12:04:01 [ERR] memberlist: Failed to send ack: write tcp 10.242.204.71:8301->10.242.7.74:55366: i/o timeout from=10.242.7.74:55366 2021/06/05 12:04:01 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:57930 2021/06/05 12:04:01 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:57018 2021/06/05 12:04:01 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:57932 2021/06/05 12:04:01 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:57056 2021/06/05 12:04:01 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:57258 2021/06/05 12:04:01 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.7.75 2021/06/05 12:04:01 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.7.75:8300) (DC: dc1) 2021/06/05 12:04:01 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 12:04:01 [WARN] yamux: failed to send ping reply: session shutdown 2021/06/05 12:04:01 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:57298 2021/06/05 12:04:01 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:58312 2021/06/05 12:04:01 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:58330 2021/06/05 12:04:01 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:58808 2021/06/05 12:04:01 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:58838 2021/06/05 12:04:01 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:59206 2021/06/05 12:04:01 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:57550 2021/06/05 12:04:01 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:57568 2021/06/05 12:04:01 [ERR] raft-net: Failed to flush response: write tcp 10.242.204.71:8300->10.242.37.70:55296: write: broken pipe 2021/06/05 12:04:11 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:59642 2021/06/05 12:05:07 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-5tc7t) 2021/06/05 12:05:10 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:43868->10.242.7.74:8301: i/o timeout 2021/06/05 12:05:10 [INFO] memberlist: Suspect dev-consul-7467bf7b46-5tc7t has failed, no acks received 2021/06/05 12:05:18 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:60786 2021/06/05 12:05:19 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:60876 2021/06/05 12:05:19 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:35106->10.242.37.70:8301: i/o timeout 2021/06/05 12:05:19 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/06/05 12:05:19 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1) 2021/06/05 12:05:19 [ERR] raft-net: Failed to flush response: write tcp 10.242.204.71:8300->10.242.37.70:59228: write: broken pipe 2021/06/05 12:06:36 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 12:06:36 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 12:06:36 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (0 peer confirmations) 2021/06/05 12:06:36 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.37.70 2021/06/05 12:06:36 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:32820 2021/06/05 12:06:36 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:32842 2021/06/05 12:06:37 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:33706 2021/06/05 12:06:37 [INFO] memberlist: Suspect dev-consul-7467bf7b46-5tc7t has failed, no acks received 2021/06/05 12:06:37 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1) 2021/06/05 12:06:37 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.37.70:8300) (DC: dc1) 2021/06/05 12:06:37 [WARN] yamux: failed to send ping reply: session shutdown 2021/06/05 12:06:37 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.37.70 2021/06/05 12:06:37 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:33298 2021/06/05 12:06:37 [WARN] yamux: failed to send ping reply: session shutdown 2021/06/05 12:06:37 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:33790 2021/06/05 12:06:37 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:33346 2021/06/05 12:06:37 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:33464 2021/06/05 12:06:37 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:33516 2021/06/05 12:06:37 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:32980 2021/06/05 12:06:37 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:33024 2021/06/05 12:06:37 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:33862 2021/06/05 12:06:37 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:33188 2021/06/05 12:06:37 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:33136 2021/06/05 12:06:37 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:33934 2021/06/05 12:06:37 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.37.70:8300) (DC: dc1) 2021/06/05 12:06:39 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 12:06:39 [ERR] raft-net: Failed to flush response: write tcp 10.242.204.71:8300->10.242.37.70:60786: write: broken pipe 2021/06/05 12:06:41 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:34044 2021/06/05 12:06:55 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:34256 2021/06/05 12:06:55 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-5tc7t) 2021/06/05 12:06:55 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:52708->10.242.7.75:8301: i/o timeout 2021/06/05 12:06:55 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/05 12:06:56 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1) 2021/06/05 12:07:17 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:51656->10.242.7.75:8302: i/o timeout 2021/06/05 12:07:17 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/06/05 12:07:28 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0) 2021/06/05 12:07:28 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:45532->10.242.7.74:8301: i/o timeout 2021/06/05 12:07:28 [INFO] memberlist: Suspect dev-consul-7467bf7b46-5tc7t has failed, no acks received 2021/06/05 12:07:29 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:34614 2021/06/05 12:07:29 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:34806 2021/06/05 12:07:29 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1) 2021/06/05 12:08:48 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:54640->10.242.7.75:8301: i/o timeout 2021/06/05 12:08:48 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/05 12:08:48 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-5tc7t) 2021/06/05 12:08:53 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2) 2021/06/05 12:08:54 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:54740->10.242.7.75:8301: i/o timeout 2021/06/05 12:08:54 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/05 12:09:07 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:36576 2021/06/05 12:09:07 [ERR] raft-net: Failed to flush response: write tcp 10.242.204.71:8300->10.242.37.70:34256: write: broken pipe 2021/06/05 12:09:17 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2) 2021/06/05 12:09:17 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1) 2021/06/05 12:09:24 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:36844 2021/06/05 12:09:27 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:47366->10.242.7.74:8301: i/o timeout 2021/06/05 12:09:27 [INFO] memberlist: Suspect dev-consul-7467bf7b46-5tc7t has failed, no acks received 2021/06/05 12:09:51 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:37012 2021/06/05 12:10:25 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/05 12:10:25 [INFO] memberlist: Marking dev-consul-7467bf7b46-5tc7t as failed, suspect timeout reached (0 peer confirmations) 2021/06/05 12:10:25 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-5tc7t 10.242.7.74 2021/06/05 12:10:25 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:37178 2021/06/05 12:10:26 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:55636->10.242.7.75:8301: i/o timeout 2021/06/05 12:10:27 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/05 12:10:27 [ERR] memberlist: Failed to send ack: write tcp 10.242.204.71:8302->10.242.37.70:40950: i/o timeout from=10.242.37.70:40950 2021/06/05 12:10:27 [ERR] memberlist: Failed to send ack: write tcp 10.242.204.71:8302->10.242.7.75:49950: i/o timeout from=10.242.7.75:49950 2021/06/05 12:10:31 [ERR] raft-net: Failed to flush response: write tcp 10.242.204.71:8300->10.242.37.70:36844: write: broken pipe 2021/06/05 12:10:31 [ERR] memberlist: Push/Pull with dev-consul-server-2.dc1 failed: dial tcp 10.242.37.70:8302: i/o timeout 2021/06/05 12:10:32 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:37190 2021/06/05 12:10:32 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:54426->10.242.7.75:8302: i/o timeout 2021/06/05 12:10:32 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/06/05 12:10:32 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/05 12:10:32 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:37982 2021/06/05 12:10:32 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:37348 2021/06/05 12:10:32 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:37534 2021/06/05 12:10:32 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:37818 2021/06/05 12:10:32 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:37670 2021/06/05 12:10:32 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-5tc7t 10.242.7.74 2021/06/05 12:10:32 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:37882 2021/06/05 12:10:32 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:37360 2021/06/05 12:10:32 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:37514 2021/06/05 12:10:32 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:38036 2021/06/05 12:10:33 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:37716 2021/06/05 12:10:32 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured 2021/06/05 12:10:33 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 12:10:40 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:55946->10.242.7.75:8301: i/o timeout 2021/06/05 12:10:40 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/05 12:10:52 [ERR] memberlist: Failed to send ack: write tcp 10.242.204.71:8301->10.242.7.74:38594: i/o timeout from=10.242.7.74:38594 2021/06/05 12:10:52 [ERR] memberlist: Failed to send ack: write tcp 10.242.204.71:8301->10.242.7.74:38576: i/o timeout from=10.242.7.74:38576 2021/06/05 12:10:52 [ERR] memberlist: Failed to send ack: write tcp 10.242.204.71:8301->10.242.7.75:56102: i/o timeout from=10.242.7.75:56102 2021/06/05 12:10:52 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:38304 2021/06/05 12:10:52 [ERR] raft-net: Failed to flush response: write tcp 10.242.204.71:8300->10.242.37.70:37982: write: broken pipe 2021/06/05 12:10:52 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-5tc7t) 2021/06/05 12:10:52 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:54698->10.242.7.75:8302: i/o timeout 2021/06/05 12:10:52 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/06/05 12:10:53 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:39504->10.242.37.70:8301: i/o timeout 2021/06/05 12:10:53 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/06/05 12:10:53 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1) 2021/06/05 12:11:11 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:38654 2021/06/05 12:11:11 [ERR] raft-net: Failed to flush response: write tcp 10.242.204.71:8300->10.242.37.70:38304: write: broken pipe 2021/06/05 12:11:11 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 12:11:15 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-5tc7t) 2021/06/05 12:11:22 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-5tc7t) 2021/06/05 12:11:22 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:48752->10.242.7.74:8301: i/o timeout 2021/06/05 12:12:49 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-5tc7t) 2021/06/05 12:13:01 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:41256->10.242.37.70:8301: i/o timeout 2021/06/05 12:13:01 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/06/05 12:13:24 [ERR] memberlist: Failed to send ack: write tcp 10.242.204.71:8301->10.242.37.70:39888: i/o timeout from=10.242.37.70:39888 2021/06/05 12:13:24 [ERR] yamux: keepalive failed: connection write timeout 2021/06/05 12:13:24 [WARN] yamux: failed to send ping reply: session shutdown 2021/06/05 12:13:28 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 12:13:28 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:40770 2021/06/05 12:13:28 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.7.74:47880 2021/06/05 12:13:29 [ERR] raft-net: Failed to flush response: write tcp 10.242.204.71:8300->10.242.37.70:38654: write: broken pipe 2021/06/05 12:13:29 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:40940 2021/06/05 12:13:32 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:56790->10.242.7.75:8302: i/o timeout 2021/06/05 12:13:32 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/06/05 12:13:32 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1) 2021/06/05 12:13:32 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:41096 2021/06/05 12:13:32 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:40942 2021/06/05 12:13:32 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:41110 2021/06/05 12:13:33 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 12:13:34 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:41244 2021/06/05 12:13:34 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:50502->10.242.7.74:8301: i/o timeout 2021/06/05 12:13:46 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0) 2021/06/05 12:13:47 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:41936->10.242.37.70:8301: i/o timeout 2021/06/05 12:13:47 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/06/05 12:13:54 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-5tc7t) 2021/06/05 12:13:55 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:50956->10.242.7.74:8301: i/o timeout 2021/06/05 12:13:55 [INFO] memberlist: Suspect dev-consul-7467bf7b46-5tc7t has failed, no acks received 2021/06/05 12:14:09 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0) 2021/06/05 12:14:13 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:58858->10.242.7.75:8301: i/o timeout 2021/06/05 12:14:13 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/05 12:14:41 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/05 12:14:41 [ERR] memberlist: Push/Pull with dev-consul-server-0 failed: write tcp 10.242.204.71:58768->10.242.7.75:8301: i/o timeout 2021/06/05 12:14:41 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:42442 2021/06/05 12:14:41 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:42496 2021/06/05 12:14:41 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:42076 2021/06/05 12:14:41 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:42090 2021/06/05 12:14:41 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:42264 2021/06/05 12:14:41 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:42294 2021/06/05 12:14:41 [ERR] raft-net: Failed to flush response: write tcp 10.242.204.71:8300->10.242.37.70:41244: write: broken pipe 2021/06/05 12:15:05 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2) 2021/06/05 12:15:05 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:43056->10.242.37.70:8301: i/o timeout 2021/06/05 12:15:35 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 12:15:38 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 12:15:38 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:43282 2021/06/05 12:15:38 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:43288 2021/06/05 12:15:38 [WARN] raft: Heartbeat timeout from "10.242.37.70:8300" reached, starting election 2021/06/05 12:15:39 [INFO] raft: Node at 10.242.204.71:8300 [Candidate] entering Candidate state in term 10 2021/06/05 12:15:39 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:43458 2021/06/05 12:15:39 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:43434 2021/06/05 12:15:39 [ERR] memberlist: Push/Pull with dev-consul-server-2 failed: EOF 2021/06/05 12:15:38 [WARN] yamux: failed to send ping reply: session shutdown 2021/06/05 12:15:43 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 12:15:48 [INFO] raft: Node at 10.242.204.71:8300 [Follower] entering Follower state (Leader: "") 2021/06/05 12:15:46 [ERR] agent: failed to sync remote state: No cluster leader 2021/06/05 12:15:46 [ERR] agent: Coordinate update error: No cluster leader 2021/06/05 12:15:47 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:34810->10.242.37.70:8302: i/o timeout 2021/06/05 12:15:48 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received 2021/06/05 12:15:48 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1) 2021/06/05 12:15:48 [ERR] raft-net: Failed to flush response: write tcp 10.242.204.71:8300->10.242.37.70:42442: write: broken pipe 2021/06/05 12:15:54 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:43602 2021/06/05 12:15:54 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:43648 2021/06/05 12:15:58 [WARN] raft: Heartbeat timeout from "" reached, starting election 2021/06/05 12:15:58 [INFO] raft: Node at 10.242.204.71:8300 [Candidate] entering Candidate state in term 11 2021/06/05 12:16:00 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:43810 2021/06/05 12:16:02 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.79.8:33054 2021/06/05 12:16:03 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.79.8:32996 2021/06/05 12:16:04 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:43764 2021/06/05 12:16:06 [ERR] memberlist: Failed to push local state: write tcp 10.242.204.71:8301->10.242.37.70:42872: i/o timeout from=10.242.37.70:42872 2021/06/05 12:16:06 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:60426->10.242.7.75:8301: i/o timeout 2021/06/05 12:16:06 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/05 12:16:12 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:43904 2021/06/05 12:16:12 [WARN] memberlist: Was able to connect to dev-consul-server-0.dc1 but other probes failed, network may be misconfigured 2021/06/05 12:16:14 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-5tc7t but other probes failed, network may be misconfigured 2021/06/05 12:16:14 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0) 2021/06/05 12:16:14 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:44074 2021/06/05 12:16:16 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:44108 2021/06/05 12:16:16 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0) 2021/06/05 12:16:16 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1) 2021/06/05 12:16:16 [INFO] raft: Node at 10.242.204.71:8300 [Follower] entering Follower state (Leader: "10.242.37.70:8300") 2021/06/05 12:17:02 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:53194->10.242.7.74:8301: i/o timeout 2021/06/05 12:17:03 [ERR] raft-net: Failed to flush response: write tcp 10.242.204.71:8300->10.242.37.70:44074: write: broken pipe 2021/06/05 12:17:04 [INFO] memberlist: Suspect dev-consul-7467bf7b46-5tc7t has failed, no acks received 2021/06/05 12:17:05 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 12:17:05 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0) 2021/06/05 12:17:07 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:44678 2021/06/05 12:17:08 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:44854 2021/06/05 12:17:10 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:44648 2021/06/05 12:17:10 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:44810 2021/06/05 12:17:10 [ERR] agent: Coordinate update error: rpc error making call: stream closed 2021/06/05 12:17:12 [ERR] yamux: Failed to write header: write tcp 10.242.204.71:59782->10.242.37.70:8300: write: broken pipe 2021/06/05 12:17:12 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:35762->10.242.37.70:8302: i/o timeout 2021/06/05 12:17:12 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received 2021/06/05 12:17:12 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:44972 2021/06/05 12:17:12 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:33098->10.242.7.75:8301: i/o timeout 2021/06/05 12:17:12 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/05 12:17:12 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1) 2021/06/05 12:17:12 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 12:22:28 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 12:22:28 [ERR] memberlist: Failed to send ack: write tcp 10.242.204.71:8301->10.242.7.75:39020: i/o timeout from=10.242.7.75:39020 2021/06/05 12:22:28 [WARN] yamux: failed to send ping reply: session shutdown 2021/06/05 12:22:28 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 12:22:28 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 12:22:28 [ERR] yamux: Failed to write header: write tcp 10.242.204.71:8300->10.242.37.70:33292: use of closed network connection 2021/06/05 12:22:28 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 12:22:36 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/05 12:22:36 [ERR] agent: Coordinate update error: rpc error making call: stream closed 2021/06/05 12:22:36 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1) 2021/06/05 12:22:38 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 12:22:39 [ERR] memberlist: Failed to push local state: write tcp 10.242.204.71:8301->10.242.7.75:40486: i/o timeout from=10.242.7.75:40486 2021/06/05 12:22:39 [ERR] memberlist: Failed to push local state: write tcp 10.242.204.71:8301->10.242.37.70:49184: i/o timeout from=10.242.37.70:49184 2021/06/05 12:22:39 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:35532->10.242.7.75:8302: i/o timeout 2021/06/05 12:22:39 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/06/05 12:22:39 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:49064 2021/06/05 12:22:39 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:50920 2021/06/05 12:22:39 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:49968 2021/06/05 12:22:39 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:50050 2021/06/05 12:22:39 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:50154 2021/06/05 12:22:39 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:50252 2021/06/05 12:22:39 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:50738 2021/06/05 12:22:39 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:50326 2021/06/05 12:22:39 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:50468 2021/06/05 12:22:39 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:50524 2021/06/05 12:22:39 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:50696 2021/06/05 12:22:39 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:49388 2021/06/05 12:22:39 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:49090 2021/06/05 12:22:39 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:49228 2021/06/05 12:22:39 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:49264 2021/06/05 12:22:39 [ERR] memberlist: Failed to push local state: write tcp 10.242.204.71:8301->10.242.7.74:50680: i/o timeout from=10.242.7.74:50680 2021/06/05 12:22:39 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:49530 2021/06/05 12:22:39 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:49654 2021/06/05 12:22:39 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:49698 2021/06/05 12:22:39 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:49796 2021/06/05 12:22:39 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:49856 2021/06/05 12:22:39 [ERR] raft-net: Failed to flush response: write tcp 10.242.204.71:8300->10.242.37.70:44972: write: broken pipe 2021/06/05 12:22:39 [WARN] yamux: failed to send ping reply: session shutdown 2021/06/05 12:22:40 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:50978 2021/06/05 12:23:06 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:57968->10.242.7.74:8301: i/o timeout 2021/06/05 12:23:06 [INFO] memberlist: Suspect dev-consul-7467bf7b46-5tc7t has failed, no acks received 2021/06/05 12:23:20 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:49186->10.242.37.70:8301: i/o timeout 2021/06/05 12:23:32 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:36156->10.242.7.75:8302: i/o timeout 2021/06/05 12:23:32 [ERR] memberlist: Failed to send ack: write tcp 10.242.204.71:8301->10.242.7.74:52382: i/o timeout from=10.242.7.74:52382 2021/06/05 12:23:34 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 12:23:56 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/06/05 12:23:56 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/06/05 12:24:07 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:51628 2021/06/05 12:23:57 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/05 12:24:07 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:52136 2021/06/05 12:24:07 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.204.71:54694 2021/06/05 12:24:08 [INFO] memberlist: Marking dev-consul-7467bf7b46-5tc7t as failed, suspect timeout reached (0 peer confirmations) 2021/06/05 12:24:08 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:51964 2021/06/05 12:24:08 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:51806 2021/06/05 12:24:08 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:52108 2021/06/05 12:24:08 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 12:24:08 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 12:24:08 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 12:24:08 [ERR] memberlist: Push/Pull with dev-consul-server-0 failed: write tcp 10.242.204.71:37550->10.242.7.75:8301: i/o timeout 2021/06/05 12:24:08 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-5tc7t 10.242.7.74 2021/06/05 12:24:08 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:52426 2021/06/05 12:24:08 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:52266 2021/06/05 12:24:08 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:51796 2021/06/05 12:24:14 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:52310 2021/06/05 12:24:15 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:51950 2021/06/05 12:24:16 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:52476 2021/06/05 12:24:17 [ERR] raft-net: Failed to flush response: write tcp 10.242.204.71:8300->10.242.37.70:50920: write: broken pipe 2021/06/05 12:24:17 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:52688 2021/06/05 12:24:17 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-5tc7t 10.242.7.74 2021/06/05 12:24:17 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/05 12:24:39 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (0 peer confirmations) 2021/06/05 12:24:51 [ERR] memberlist: Push/Pull with dev-consul-server-2.dc1 failed: write tcp 10.242.204.71:40982->10.242.37.70:8302: i/o timeout 2021/06/05 12:24:51 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:50608->10.242.37.70:8301: i/o timeout 2021/06/05 12:25:06 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/05 12:25:06 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:53270 2021/06/05 12:25:06 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:53252 2021/06/05 12:25:06 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:52846 2021/06/05 12:25:06 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:53030 2021/06/05 12:25:06 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:53470 2021/06/05 12:25:06 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:53056 2021/06/05 12:25:06 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.37.70 2021/06/05 12:25:06 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/06/05 12:25:08 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:53484 2021/06/05 12:25:04 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/06/05 12:25:08 [ERR] memberlist: Push/Pull with dev-consul-server-0 failed: EOF 2021/06/05 12:25:08 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.37.70:8300) (DC: dc1) 2021/06/05 12:25:08 [ERR] memberlist: Failed to push local state: write tcp 10.242.204.71:8302->10.242.37.70:56256: i/o timeout from=10.242.37.70:56256 2021/06/05 12:25:15 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.7.75 2021/06/05 12:25:15 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan" 2021/06/05 12:25:15 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:42126->10.242.37.70:8302: i/o timeout 2021/06/05 12:25:15 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received 2021/06/05 12:27:10 [WARN] yamux: failed to send ping reply: session shutdown 2021/06/05 12:27:10 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/06/05 12:27:10 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.37.70 2021/06/05 12:27:10 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.37.70:8300) (DC: dc1) 2021/06/05 12:27:10 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.37.70 2021/06/05 12:27:10 [ERR] yamux: keepalive failed: session shutdown 2021/06/05 12:27:10 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan" 2021/06/05 12:27:10 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:55816 2021/06/05 12:27:10 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:53912 2021/06/05 12:27:10 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:53936 2021/06/05 12:27:10 [ERR] consul.rpc: multiplex conn accept failed: session shutdown from=10.242.37.70:54028 2021/06/05 12:27:10 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:54078 2021/06/05 12:27:11 [ERR] memberlist: Failed fallback ping: write tcp 10.242.204.71:39878->10.242.7.75:8301: i/o timeout 2021/06/05 12:27:11 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/06/05 12:27:11 [WARN] yamux: failed to send ping reply: session shutdown 2021/06/05 12:27:11 [WARN] yamux: failed to send ping reply: session shutdown 2021/06/05 12:27:11 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:54458 2021/06/05 12:27:11 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:55004 2021/06/05 12:27:11 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:54126 2021/06/05 12:27:11 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:54270 2021/06/05 12:27:11 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:54320 2021/06/05 12:27:11 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:54528 2021/06/05 12:27:11 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:54818 2021/06/05 12:27:11 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:54924 2021/06/05 12:27:11 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:55186 2021/06/05 12:27:11 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:55544 2021/06/05 12:27:11 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:55318 2021/06/05 12:27:11 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:55388 2021/06/05 12:27:11 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:55122 2021/06/05 12:27:11 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:55778 2021/06/05 12:27:11 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:55582 2021/06/05 12:27:11 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/05 12:27:11 [ERR] raft-net: Failed to flush response: write tcp 10.242.204.71:8300->10.242.37.70:53484: write: broken pipe 2021/06/05 12:27:11 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:54628 2021/06/05 12:27:11 [DEBUG] raft-net: 10.242.204.71:8300 accepted connection from: 10.242.37.70:54722 2021/06/05 12:27:11 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.7.75 2021/06/05 12:27:11 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan" 2021/06/05 12:27:11 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2) 2021/06/05 12:27:11 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.37.70 2021/06/05 12:27:11 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"