By type
+ CONSUL_BIND=
+ [ -n ]
+ CONSUL_CLIENT=
+ [ -n ]
+ CONSUL_DATA_DIR=/consul/data
+ CONSUL_CONFIG_DIR=/consul/config
+ [ -n ]
+ [ a = - ]
+ [ agent = agent ]
+ shift
+ set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
+ [ consul = consul ]
+ stat -c %u /consul/data
+ id -u consul
+ [ 100 != 100 ]
+ stat -c %u /consul/config
+ id -u consul
+ [ 100 != 100 ]
+ [ ! -z ]
+ exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
bootstrap_expect > 0: expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
Version: 'v1.0.6'
Node ID: '54ac3dd7-ebc8-ef78-c2aa-b2bd4e20208b'
Node name: 'dev-consul-server-2'
Datacenter: 'dc1' (Segment: '')
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.242.203.14 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2021/05/03 22:44:00 [INFO] raft: Initial configuration (index=0): []
2021/05/03 22:44:00 [INFO] raft: Node at 10.242.203.14:8300 [Follower] entering Follower state (Leader: "")
2021/05/03 22:44:00 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.203.14
2021/05/03 22:44:00 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.203.14
2021/05/03 22:44:00 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/05/03 22:44:00 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.203.14:8300) (DC: dc1)
2021/05/03 22:44:00 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2021/05/03 22:44:00 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2021/05/03 22:44:00 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
2021/05/03 22:44:00 [INFO] agent: started state syncer
2021/05/03 22:44:00 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer
2021/05/03 22:44:00 [INFO] agent: Joining LAN cluster...
2021/05/03 22:44:00 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/05/03 22:44:00 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 22:44:00 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/03 22:44:00 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.193.202
2021/05/03 22:44:00 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 22:44:00 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.242.8.142:8300,10.242.193.202:8300,10.242.203.14:8300
2021/05/03 22:44:00 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.193.202
2021/05/03 22:44:00 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.8.142
2021/05/03 22:44:00 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/03 22:44:00 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/03 22:44:00 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/05/03 22:44:00 [INFO] agent: (LAN) joined: 2 Err:
2021/05/03 22:44:00 [INFO] agent: Join LAN completed. Synced with 2 initial agents
2021/05/03 22:44:00 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.193.202:8300) (DC: dc1)
2021/05/03 22:44:07 [ERR] agent: failed to sync remote state: No cluster leader
2021/05/03 22:44:07 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.8.142:50486
2021/05/03 22:44:13 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.8.142:50542
2021/05/03 22:44:13 [INFO] consul: New leader elected: dev-consul-server-0
2021/05/03 22:44:16 [INFO] agent: Synced node info
==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=5361985e-3138-e245-f8fa-83d195dbf553&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2021/05/03 23:05:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:60520->10.242.8.143:8301: i/o timeout
2021/05/03 23:05:01 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:05:03 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/03 23:05:03 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:05:04 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:05:04 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-gbhmt but other probes failed, network may be misconfigured
2021/05/03 23:05:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:60864->10.242.8.143:8301: i/o timeout
2021/05/03 23:05:26 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:06:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:56302->10.242.8.142:8301: i/o timeout
2021/05/03 23:06:02 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:06:04 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-gbhmt)
2021/05/03 23:06:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:33194->10.242.8.143:8301: i/o timeout
2021/05/03 23:06:08 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:06:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:33776->10.242.8.143:8301: i/o timeout
2021/05/03 23:06:51 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:06:52 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-gbhmt)
2021/05/03 23:06:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:33872->10.242.8.143:8301: i/o timeout
2021/05/03 23:06:58 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:07:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:57254->10.242.8.142:8301: i/o timeout
2021/05/03 23:07:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:34084->10.242.8.143:8301: i/o timeout
2021/05/03 23:07:15 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:08:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:35350->10.242.8.143:8301: i/o timeout
2021/05/03 23:08:55 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:08:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:35388->10.242.8.143:8301: i/o timeout
2021/05/03 23:08:58 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:09:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:58622->10.242.8.142:8301: i/o timeout
2021/05/03 23:09:01 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:09:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:35430->10.242.8.143:8301: i/o timeout
2021/05/03 23:09:03 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:09:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:58698->10.242.8.142:8301: i/o timeout
2021/05/03 23:09:05 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:09:05 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (1 peer confirmations)
2021/05/03 23:09:05 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:09:06 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:09:07 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-gbhmt but other probes failed, network may be misconfigured
2021/05/03 23:09:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:35538->10.242.8.143:8301: i/o timeout
2021/05/03 23:09:09 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:09:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:35572->10.242.8.143:8301: i/o timeout
2021/05/03 23:09:12 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:09:13 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/03 23:09:13 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:09:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:35608->10.242.8.143:8301: i/o timeout
2021/05/03 23:09:15 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:09:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:58844->10.242.8.142:8301: i/o timeout
2021/05/03 23:09:18 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:09:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:58884->10.242.8.142:8301: i/o timeout
2021/05/03 23:09:21 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:09:21 [WARN] raft: Rejecting vote request from 10.242.193.202:8300 since we have a leader: 10.242.8.142:8300
2021/05/03 23:09:21 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:55162
2021/05/03 23:09:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:58940->10.242.8.142:8301: i/o timeout
2021/05/03 23:09:23 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:09:24 [WARN] raft: Heartbeat timeout from "10.242.8.142:8300" reached, starting election
2021/05/03 23:09:24 [INFO] raft: Node at 10.242.203.14:8300 [Candidate] entering Candidate state in term 3
2021/05/03 23:09:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:50488->10.242.8.142:8302: i/o timeout
2021/05/03 23:09:25 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:09:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:58982->10.242.8.142:8301: i/o timeout
2021/05/03 23:09:26 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:09:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:58994->10.242.8.142:8301: i/o timeout
2021/05/03 23:09:28 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:09:28 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations)
2021/05/03 23:09:28 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/03 23:09:28 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:09:30 [INFO] raft: Node at 10.242.203.14:8300 [Follower] entering Follower state (Leader: "")
2021/05/03 23:09:30 [INFO] consul: New leader elected: dev-consul-server-1
2021/05/03 23:09:30 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/03 23:09:30 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:55324
2021/05/03 23:09:34 [ERR] raft: Failed to make RequestVote RPC to {Voter 7de1a64f-08d1-f432-1ba4-3ff4aa5184c9 10.242.8.142:8300}: read tcp 10.242.203.14:49202->10.242.8.142:8300: i/o timeout
2021/05/03 23:09:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:50618->10.242.8.142:8302: i/o timeout
2021/05/03 23:09:35 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:09:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:50752->10.242.8.142:8302: i/o timeout
2021/05/03 23:09:45 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:09:45 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/03 23:09:45 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.8.142:50416
2021/05/03 23:09:45 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/03 23:09:45 [ERR] agent: Coordinate update error: rpc error making call: EOF
2021/05/03 23:09:45 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: rpc error making call: EOF from=10.242.87.8:50836
2021/05/03 23:09:54 [ERR] memberlist: failed to receive: read tcp 10.242.203.14:8301->10.242.8.142:53562: i/o timeout from=10.242.8.142:53562
2021/05/03 23:09:54 [ERR] memberlist: Failed to send error: write tcp 10.242.203.14:8301->10.242.8.142:53562: i/o timeout from=10.242.8.142:53562
2021/05/03 23:09:55 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/03 23:09:55 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.8.142
2021/05/03 23:09:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:50826->10.242.8.142:8302: i/o timeout
2021/05/03 23:09:55 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/03 23:09:55 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:10:10 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/03 23:10:20 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/05/03 23:10:29 [ERR] memberlist: failed to receive: read tcp 10.242.203.14:8302->10.242.8.142:34718: i/o timeout from=10.242.8.142:34718
2021/05/03 23:10:29 [ERR] memberlist: Failed to send error: write tcp 10.242.203.14:8302->10.242.8.142:34718: i/o timeout from=10.242.8.142:34718
2021/05/03 23:10:50 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/03 23:11:19 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/03 23:11:19 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:11:19 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.8.142
2021/05/03 23:11:19 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/03 23:11:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:60520->10.242.8.142:8301: i/o timeout
2021/05/03 23:11:23 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:11:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:60546->10.242.8.142:8301: i/o timeout
2021/05/03 23:11:26 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:11:27 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (0 peer confirmations)
2021/05/03 23:11:27 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/03 23:11:27 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:11:28 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:11:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:37400->10.242.8.143:8301: i/o timeout
2021/05/03 23:11:30 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:11:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:37422->10.242.8.143:8301: i/o timeout
2021/05/03 23:11:32 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:11:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:37462->10.242.8.143:8301: i/o timeout
2021/05/03 23:11:35 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:11:35 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/05/03 23:11:36 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/03 23:11:36 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:11:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:60704->10.242.8.142:8301: i/o timeout
2021/05/03 23:11:39 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:11:40 [ERR] memberlist: failed to receive: read tcp 10.242.203.14:8301->10.242.8.143:47148: i/o timeout from=10.242.8.143:47148
2021/05/03 23:11:40 [ERR] memberlist: Failed to send error: write tcp 10.242.203.14:8301->10.242.8.143:47148: i/o timeout from=10.242.8.143:47148
2021/05/03 23:11:41 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (1 peer confirmations)
2021/05/03 23:11:41 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:11:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:37530->10.242.8.143:8301: i/o timeout
2021/05/03 23:11:42 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:11:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:60808->10.242.8.142:8301: i/o timeout
2021/05/03 23:11:45 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:11:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:60842->10.242.8.142:8301: i/o timeout
2021/05/03 23:11:47 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:11:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:60876->10.242.8.142:8301: i/o timeout
2021/05/03 23:11:49 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:11:50 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations)
2021/05/03 23:11:50 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/03 23:11:50 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:11:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:52482->10.242.8.142:8302: i/o timeout
2021/05/03 23:11:55 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:11:58 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/03 23:11:58 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.8.143:48558
2021/05/03 23:12:00 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/03 23:12:05 [WARN] memberlist: Was able to connect to dev-consul-server-0.dc1 but other probes failed, network may be misconfigured
2021/05/03 23:12:12 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/03 23:12:12 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.8.142
2021/05/03 23:12:12 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/03 23:12:29 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.8.142
2021/05/03 23:12:29 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/03 23:12:29 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/03 23:12:29 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:12:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:33206->10.242.8.142:8301: i/o timeout
2021/05/03 23:12:32 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:12:33 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:12:34 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/05/03 23:12:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:38272->10.242.8.143:8301: i/o timeout
2021/05/03 23:12:35 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:12:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:38570->10.242.8.143:8301: i/o timeout
2021/05/03 23:12:53 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:12:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:38634->10.242.8.143:8301: i/o timeout
2021/05/03 23:12:58 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:13:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:38654->10.242.8.143:8301: i/o timeout
2021/05/03 23:13:01 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:13:02 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:13:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:38702->10.242.8.143:8301: i/o timeout
2021/05/03 23:13:03 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:13:05 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:13:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:38774->10.242.8.143:8301: i/o timeout
2021/05/03 23:13:08 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:13:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:38784->10.242.8.143:8301: i/o timeout
2021/05/03 23:13:10 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:13:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:39008->10.242.8.143:8301: i/o timeout
2021/05/03 23:13:24 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:13:26 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:13:41 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:13:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:39204->10.242.8.143:8301: i/o timeout
2021/05/03 23:13:42 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:13:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:39252->10.242.8.143:8301: i/o timeout
2021/05/03 23:13:44 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:13:46 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/03 23:13:46 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:13:48 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:13:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:39384->10.242.8.143:8301: i/o timeout
2021/05/03 23:14:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:40178->10.242.8.143:8301: i/o timeout
2021/05/03 23:15:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:35168->10.242.8.142:8301: i/o timeout
2021/05/03 23:15:00 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:15:02 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/05/03 23:15:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:40280->10.242.8.143:8301: i/o timeout
2021/05/03 23:15:07 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:15:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:35286->10.242.8.142:8301: i/o timeout
2021/05/03 23:15:10 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:15:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:40330->10.242.8.143:8301: i/o timeout
2021/05/03 23:15:12 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:15:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:35358->10.242.8.142:8301: i/o timeout
2021/05/03 23:15:14 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:15:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:55122->10.242.8.142:8302: i/o timeout
2021/05/03 23:15:15 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:15:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:35368->10.242.8.142:8301: i/o timeout
2021/05/03 23:15:16 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:15:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:40414->10.242.8.143:8301: i/o timeout
2021/05/03 23:15:18 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:15:18 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (1 peer confirmations)
2021/05/03 23:15:18 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:15:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:35438->10.242.8.142:8301: i/o timeout
2021/05/03 23:15:21 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:15:21 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations)
2021/05/03 23:15:21 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/03 23:15:21 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:15:22 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/03 23:15:22 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:15:22 [WARN] raft: Rejecting vote request from 10.242.8.142:8300 since we have a leader: 10.242.193.202:8300
2021/05/03 23:15:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:35496->10.242.8.142:8301: i/o timeout
2021/05/03 23:15:24 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:15:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:35520->10.242.8.142:8301: i/o timeout
2021/05/03 23:15:26 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:15:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:35542->10.242.8.142:8301: i/o timeout
2021/05/03 23:15:28 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:15:31 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/05/03 23:15:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:35584->10.242.8.142:8301: i/o timeout
2021/05/03 23:15:32 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:15:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:35628->10.242.8.142:8301: i/o timeout
2021/05/03 23:15:35 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:15:35 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations)
2021/05/03 23:15:35 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/03 23:15:35 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:15:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:35644->10.242.8.142:8301: i/o timeout
2021/05/03 23:15:37 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:15:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:55450->10.242.8.142:8302: i/o timeout
2021/05/03 23:15:40 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:15:40 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/03 23:15:41 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-gbhmt)
2021/05/03 23:15:41 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:15:41 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/03 23:15:41 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:15:42 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/05/03 23:16:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:41340->10.242.8.143:8301: i/o timeout
2021/05/03 23:16:26 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:16:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:41370->10.242.8.143:8301: i/o timeout
2021/05/03 23:16:29 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:16:30 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/03 23:16:30 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:16:38 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-gbhmt)
2021/05/03 23:16:38 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:16:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:41604->10.242.8.143:8301: i/o timeout
2021/05/03 23:16:47 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:16:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:36622->10.242.8.142:8301: i/o timeout
2021/05/03 23:16:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:41690->10.242.8.143:8301: i/o timeout
2021/05/03 23:16:53 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:16:56 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-gbhmt but other probes failed, network may be misconfigured
2021/05/03 23:16:57 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/03 23:16:57 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:17:00 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:17:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:41780->10.242.8.143:8301: i/o timeout
2021/05/03 23:17:02 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:17:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:36824->10.242.8.142:8301: i/o timeout
2021/05/03 23:17:05 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:17:06 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/03 23:17:06 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:17:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:36866->10.242.8.142:8301: i/o timeout
2021/05/03 23:17:08 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:17:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:36894->10.242.8.142:8301: i/o timeout
2021/05/03 23:17:11 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:17:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:36958->10.242.8.142:8301: i/o timeout
2021/05/03 23:17:14 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:17:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:56722->10.242.8.142:8302: i/o timeout
2021/05/03 23:17:15 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:17:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:36984->10.242.8.142:8301: i/o timeout
2021/05/03 23:17:16 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:17:16 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations)
2021/05/03 23:17:16 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/03 23:17:16 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:17:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:56790->10.242.8.142:8302: i/o timeout
2021/05/03 23:17:25 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:17:35 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/03 23:17:35 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:17:35 [WARN] raft: Rejecting vote request from 10.242.8.142:8300 since we have a leader: 10.242.193.202:8300
2021/05/03 23:17:36 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:18:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:42684->10.242.8.143:8301: i/o timeout
2021/05/03 23:18:08 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:18:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:37674->10.242.8.142:8301: i/o timeout
2021/05/03 23:18:10 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:18:10 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/03 23:18:10 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:18:11 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/03 23:18:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:37732->10.242.8.142:8301: i/o timeout
2021/05/03 23:18:13 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:18:17 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:18:17 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/05/03 23:18:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:37796->10.242.8.142:8301: i/o timeout
2021/05/03 23:18:20 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:18:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:42998->10.242.8.143:8301: i/o timeout
2021/05/03 23:18:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:43056->10.242.8.143:8301: i/o timeout
2021/05/03 23:18:38 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:19:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:43968->10.242.8.143:8301: i/o timeout
2021/05/03 23:19:51 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:19:54 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/05/03 23:19:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:44042->10.242.8.143:8301: i/o timeout
2021/05/03 23:19:56 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:19:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:44062->10.242.8.143:8301: i/o timeout
2021/05/03 23:19:58 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:20:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:39050->10.242.8.142:8301: i/o timeout
2021/05/03 23:20:00 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:20:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:44102->10.242.8.143:8301: i/o timeout
2021/05/03 23:20:02 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:20:02 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (1 peer confirmations)
2021/05/03 23:20:02 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:20:02 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/03 23:20:02 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:20:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:39112->10.242.8.142:8301: i/o timeout
2021/05/03 23:20:04 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:20:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:58890->10.242.8.142:8302: i/o timeout
2021/05/03 23:20:05 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:20:14 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/03 23:20:14 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.8.143:55090
2021/05/03 23:20:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:59012->10.242.8.142:8302: i/o timeout
2021/05/03 23:20:15 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:20:17 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/03 23:20:18 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/03 23:20:18 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:20:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:39324->10.242.8.142:8301: i/o timeout
2021/05/03 23:20:21 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:20:21 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-gbhmt)
2021/05/03 23:20:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:39370->10.242.8.142:8301: i/o timeout
2021/05/03 23:20:24 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:20:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:39408->10.242.8.142:8301: i/o timeout
2021/05/03 23:20:27 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:20:28 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/05/03 23:20:30 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:20:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:39456->10.242.8.142:8301: i/o timeout
2021/05/03 23:20:30 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:20:31 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/03 23:20:31 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:20:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:39466->10.242.8.142:8301: i/o timeout
2021/05/03 23:20:32 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:20:34 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/03 23:20:34 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:20:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:44530->10.242.8.143:8301: i/o timeout
2021/05/03 23:20:35 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:20:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:39560->10.242.8.142:8301: i/o timeout
2021/05/03 23:20:38 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:20:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:44614->10.242.8.143:8301: i/o timeout
2021/05/03 23:20:40 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:20:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:44678->10.242.8.143:8301: i/o timeout
2021/05/03 23:20:43 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:20:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:39608->10.242.8.142:8301: i/o timeout
2021/05/03 23:20:44 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (1 peer confirmations)
2021/05/03 23:20:44 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:20:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:44694->10.242.8.143:8301: i/o timeout
2021/05/03 23:20:46 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:20:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:39788->10.242.8.142:8301: i/o timeout
2021/05/03 23:20:52 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:20:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:39848->10.242.8.142:8301: i/o timeout
2021/05/03 23:20:56 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:20:57 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/03 23:20:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:39866->10.242.8.142:8301: i/o timeout
2021/05/03 23:20:58 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:21:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:39906->10.242.8.142:8301: i/o timeout
2021/05/03 23:21:02 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:21:03 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations)
2021/05/03 23:21:03 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/03 23:21:03 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:21:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:39964->10.242.8.142:8301: i/o timeout
2021/05/03 23:21:04 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:21:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:59726->10.242.8.142:8302: i/o timeout
2021/05/03 23:21:05 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:21:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:59936->10.242.8.142:8302: i/o timeout
2021/05/03 23:21:20 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:21:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:60082->10.242.8.142:8302: i/o timeout
2021/05/03 23:21:30 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:21:33 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.8.142
2021/05/03 23:21:33 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/03 23:21:37 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/03 23:21:53 [ERR] memberlist: failed to receive: read tcp 10.242.203.14:8302->10.242.8.142:47896: i/o timeout from=10.242.8.142:47896
2021/05/03 23:21:53 [ERR] memberlist: Failed to send error: write tcp 10.242.203.14:8302->10.242.8.142:47896: i/o timeout from=10.242.8.142:47896
2021/05/03 23:22:09 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/03 23:22:09 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:22:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:40944->10.242.8.142:8301: i/o timeout
2021/05/03 23:22:11 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:22:11 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.8.142
2021/05/03 23:22:11 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/03 23:22:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:40994->10.242.8.142:8301: i/o timeout
2021/05/03 23:22:13 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:22:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:41020->10.242.8.142:8301: i/o timeout
2021/05/03 23:22:15 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:22:15 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (0 peer confirmations)
2021/05/03 23:22:15 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/03 23:22:15 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:22:17 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/03 23:22:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:60930->10.242.8.142:8302: i/o timeout
2021/05/03 23:22:25 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:22:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:60996->10.242.8.142:8302: i/o timeout
2021/05/03 23:22:35 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:22:47 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/03 23:22:47 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.8.142
2021/05/03 23:22:47 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/03 23:22:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:33046->10.242.8.142:8302: i/o timeout
2021/05/03 23:22:50 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:22:57 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/03 23:23:00 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.8.142:8302
2021/05/03 23:23:03 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/03 23:23:03 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:23:03 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/05/03 23:23:03 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/05/03 23:23:03 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.193.202
2021/05/03 23:23:03 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.193.202:8300) (DC: dc1)
2021/05/03 23:23:03 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.8.142
2021/05/03 23:23:03 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/03 23:23:03 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.193.202
2021/05/03 23:23:03 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.193.202:8300) (DC: dc1)
2021/05/03 23:23:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:41718->10.242.8.142:8301: i/o timeout
2021/05/03 23:23:05 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:23:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:41756->10.242.8.142:8301: i/o timeout
2021/05/03 23:23:07 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:23:07 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:23:09 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (0 peer confirmations)
2021/05/03 23:23:09 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/03 23:23:09 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:23:10 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-gbhmt but other probes failed, network may be misconfigured
2021/05/03 23:23:10 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/03 23:23:10 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:23:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:46890->10.242.8.143:8301: i/o timeout
2021/05/03 23:23:13 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:23:14 [WARN] raft: Rejecting vote request from 10.242.8.142:8300 since we have a leader: 10.242.193.202:8300
2021/05/03 23:23:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:47228->10.242.8.143:8301: i/o timeout
2021/05/03 23:23:38 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:23:42 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/03 23:23:42 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:23:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:47272->10.242.8.143:8301: i/o timeout
2021/05/03 23:23:42 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:23:44 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:23:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:47358->10.242.8.143:8301: i/o timeout
2021/05/03 23:23:47 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:23:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:47388->10.242.8.143:8301: i/o timeout
2021/05/03 23:23:50 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:23:51 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/03 23:23:51 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:23:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:42388->10.242.8.142:8301: i/o timeout
2021/05/03 23:23:53 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:23:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:42476->10.242.8.142:8301: i/o timeout
2021/05/03 23:23:56 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:23:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:42500->10.242.8.142:8301: i/o timeout
2021/05/03 23:23:58 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:24:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:42510->10.242.8.142:8301: i/o timeout
2021/05/03 23:24:00 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:24:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:42574->10.242.8.142:8301: i/o timeout
2021/05/03 23:24:03 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:24:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:34122->10.242.8.142:8302: i/o timeout
2021/05/03 23:24:05 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:24:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:42590->10.242.8.142:8301: i/o timeout
2021/05/03 23:24:05 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:24:07 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/03 23:24:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:42636->10.242.8.142:8301: i/o timeout
2021/05/03 23:24:08 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:24:09 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations)
2021/05/03 23:24:09 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/03 23:24:09 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:24:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:42662->10.242.8.142:8301: i/o timeout
2021/05/03 23:24:10 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:24:12 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:24:12 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/03 23:24:12 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:24:12 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/05/03 23:24:56 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-gbhmt)
2021/05/03 23:26:27 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-wjl58 10.242.95.21
2021/05/03 23:28:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:51246->10.242.8.143:8301: i/o timeout
2021/05/03 23:28:14 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:28:24 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/05/03 23:28:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:51412->10.242.8.143:8301: i/o timeout
2021/05/03 23:28:25 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:28:29 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-gbhmt but other probes failed, network may be misconfigured
2021/05/03 23:28:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:51500->10.242.8.143:8301: i/o timeout
2021/05/03 23:28:31 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:28:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:51604->10.242.8.143:8301: i/o timeout
2021/05/03 23:28:38 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:29:06 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-gbhmt but other probes failed, network may be misconfigured
2021/05/03 23:29:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:52246->10.242.8.143:8301: i/o timeout
2021/05/03 23:29:24 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:33:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:56360->10.242.8.143:8301: i/o timeout
2021/05/03 23:33:57 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:34:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:56614->10.242.8.143:8301: i/o timeout
2021/05/03 23:34:11 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:34:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:56702->10.242.8.143:8301: i/o timeout
2021/05/03 23:34:16 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:34:20 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/03 23:34:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:51694->10.242.8.142:8301: i/o timeout
2021/05/03 23:34:20 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:34:20 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:34:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:51780->10.242.8.142:8301: i/o timeout
2021/05/03 23:34:24 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:34:24 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2021/05/03 23:34:24 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/03 23:34:24 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:34:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:43360->10.242.8.142:8302: i/o timeout
2021/05/03 23:34:25 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:34:25 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/05/03 23:34:25 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/03 23:34:25 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:34:27 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:34:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:56930->10.242.8.143:8301: i/o timeout
2021/05/03 23:34:30 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:34:44 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/05/03 23:34:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:57196->10.242.8.143:8301: i/o timeout
2021/05/03 23:34:47 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:34:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:57342->10.242.8.143:8301: i/o timeout
2021/05/03 23:34:54 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:34:56 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2021/05/03 23:34:56 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/03 23:34:56 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:34:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:57382->10.242.8.143:8301: i/o timeout
2021/05/03 23:34:57 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:34:58 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/03 23:34:58 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:34:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:57418->10.242.8.143:8301: i/o timeout
2021/05/03 23:34:59 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:35:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:43936->10.242.8.142:8302: i/o timeout
2021/05/03 23:35:00 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:35:01 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/03 23:35:01 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:35:01 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/05/03 23:35:04 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:35:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:57752->10.242.8.143:8301: i/o timeout
2021/05/03 23:35:18 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:35:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:58002->10.242.8.143:8301: i/o timeout
2021/05/03 23:35:33 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:35:35 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/03 23:35:35 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:35:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:53070->10.242.8.142:8301: i/o timeout
2021/05/03 23:35:38 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:35:39 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:35:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:58244->10.242.8.143:8301: i/o timeout
2021/05/03 23:35:46 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:35:49 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/03 23:35:49 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:35:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:53236->10.242.8.142:8301: i/o timeout
2021/05/03 23:35:50 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:35:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:53322->10.242.8.142:8301: i/o timeout
2021/05/03 23:35:54 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:35:54 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2021/05/03 23:35:54 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/03 23:35:54 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:35:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:44900->10.242.8.142:8302: i/o timeout
2021/05/03 23:35:55 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:36:00 [WARN] raft: Rejecting vote request from 10.242.8.142:8300 since we have a leader: 10.242.193.202:8300
2021/05/03 23:36:03 [ERR] yamux: keepalive failed: session shutdown
2021/05/03 23:36:03 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/03 23:36:03 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:36:07 [WARN] raft: Rejecting vote request from 10.242.8.142:8300 since we have a leader: 10.242.193.202:8300
2021/05/03 23:36:07 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-gbhmt)
2021/05/03 23:36:10 [INFO] consul: New leader elected: dev-consul-server-1
2021/05/03 23:36:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:52152
2021/05/03 23:36:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:53754->10.242.8.142:8301: i/o timeout
2021/05/03 23:36:12 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:36:15 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/05/03 23:36:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:53888->10.242.8.142:8301: i/o timeout
2021/05/03 23:36:18 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:36:18 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/05/03 23:36:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:53910->10.242.8.142:8301: i/o timeout
2021/05/03 23:36:21 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:36:22 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2021/05/03 23:36:22 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/03 23:36:22 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:36:30 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:36:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:45678->10.242.8.142:8302: i/o timeout
2021/05/03 23:36:30 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:36:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:59216->10.242.8.143:8301: i/o timeout
2021/05/03 23:36:31 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:36:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:59262->10.242.8.143:8301: i/o timeout
2021/05/03 23:36:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:36:35 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/03 23:36:35 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:36:42 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/03 23:36:42 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/03 23:36:42 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:36:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:54514->10.242.8.142:8301: i/o timeout
2021/05/03 23:36:47 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:36:47 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/03 23:36:47 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:36:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:46090->10.242.8.142:8302: i/o timeout
2021/05/03 23:36:50 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:36:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:54540->10.242.8.142:8301: i/o timeout
2021/05/03 23:36:50 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:36:53 [ERR] memberlist: Push/Pull with dev-consul-server-0 failed: read tcp 10.242.203.14:54436->10.242.8.142:8301: i/o timeout
2021/05/03 23:37:00 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/03 23:37:00 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.8.142
2021/05/03 23:37:00 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/03 23:37:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:46334->10.242.8.142:8302: i/o timeout
2021/05/03 23:37:05 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:37:08 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.8.142
2021/05/03 23:37:08 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/03 23:37:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:46510->10.242.8.142:8302: i/o timeout
2021/05/03 23:37:15 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:37:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:46588->10.242.8.142:8302: i/o timeout
2021/05/03 23:37:25 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:37:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:46858->10.242.8.142:8302: i/o timeout
2021/05/03 23:37:35 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:37:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:47036->10.242.8.142:8302: i/o timeout
2021/05/03 23:37:45 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:37:45 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/03 23:37:45 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.8.142
2021/05/03 23:37:46 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/03 23:37:48 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/03 23:37:52 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/03 23:38:26 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.8.142
2021/05/03 23:38:26 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/03 23:38:32 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/03 23:38:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:47932->10.242.8.142:8302: i/o timeout
2021/05/03 23:38:35 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:38:42 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/05/03 23:38:42 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/03 23:38:42 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:38:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:56636->10.242.8.142:8301: i/o timeout
2021/05/03 23:38:47 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:38:47 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/03 23:38:47 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:38:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:48310->10.242.8.142:8302: i/o timeout
2021/05/03 23:38:55 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:39:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:48470->10.242.8.142:8302: i/o timeout
2021/05/03 23:39:05 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:39:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:48640->10.242.8.142:8302: i/o timeout
2021/05/03 23:39:15 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:39:22 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/03 23:39:22 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.8.142
2021/05/03 23:39:22 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/03 23:39:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:48822->10.242.8.142:8302: i/o timeout
2021/05/03 23:39:25 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:39:33 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.8.142:8302
2021/05/03 23:39:42 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/03 23:39:42 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:39:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:34432->10.242.8.143:8301: i/o timeout
2021/05/03 23:39:45 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:39:48 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:39:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:34492->10.242.8.143:8301: i/o timeout
2021/05/03 23:39:49 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:40:11 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/05/03 23:40:21 [ERR] memberlist: failed to receive: read tcp 10.242.203.14:8301->10.242.8.142:57388: i/o timeout from=10.242.8.142:57388
2021/05/03 23:40:21 [ERR] memberlist: Failed to send error: write tcp 10.242.203.14:8301->10.242.8.142:57388: i/o timeout from=10.242.8.142:57388
2021/05/03 23:40:33 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.8.142:8302
2021/05/03 23:40:42 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/03 23:40:52 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.8.142
2021/05/03 23:40:52 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/03 23:41:07 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/03 23:41:07 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.8.142:55412
2021/05/03 23:41:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:50714->10.242.8.142:8302: i/o timeout
2021/05/03 23:41:10 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:41:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:50808->10.242.8.142:8302: i/o timeout
2021/05/03 23:41:20 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:41:28 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.8.142
2021/05/03 23:41:28 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/03 23:42:22 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/03 23:43:03 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.8.142:8302
2021/05/03 23:43:32 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/03 23:43:52 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.8.142
2021/05/03 23:43:52 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/03 23:44:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:54300->10.242.8.142:8302: i/o timeout
2021/05/03 23:44:00 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:44:12 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/03 23:44:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:54624->10.242.8.142:8302: i/o timeout
2021/05/03 23:44:15 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:44:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:54748->10.242.8.142:8302: i/o timeout
2021/05/03 23:44:25 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:44:30 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/03 23:44:30 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.8.142
2021/05/03 23:44:30 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/03 23:44:33 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.8.142:8302
2021/05/03 23:44:50 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.8.142
2021/05/03 23:44:50 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/05/03 23:44:50 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/03 23:44:50 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/03 23:44:50 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:44:56 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:44:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:40904->10.242.8.143:8301: i/o timeout
2021/05/03 23:44:58 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/03 23:45:02 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/03 23:45:02 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/03 23:45:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:35910->10.242.8.142:8301: i/o timeout
2021/05/03 23:45:02 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:45:02 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/03 23:45:02 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:45:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:55792->10.242.8.142:8302: i/o timeout
2021/05/03 23:45:05 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:45:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:55898->10.242.8.142:8302: i/o timeout
2021/05/03 23:45:15 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:45:22 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/03 23:45:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:56358->10.242.8.142:8302: i/o timeout
2021/05/03 23:45:30 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:45:32 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/03 23:45:32 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.8.142
2021/05/03 23:45:32 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/03 23:45:32 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.8.142
2021/05/03 23:45:32 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/03 23:45:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:56702->10.242.8.142:8302: i/o timeout
2021/05/03 23:45:45 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:45:54 [ERR] memberlist: failed to receive: read tcp 10.242.203.14:8301->10.242.8.143:51678: i/o timeout from=10.242.8.143:51678
2021/05/03 23:45:54 [ERR] memberlist: Failed to send error: write tcp 10.242.203.14:8301->10.242.8.143:51678: i/o timeout from=10.242.8.143:51678
2021/05/03 23:45:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:56820->10.242.8.142:8302: i/o timeout
2021/05/03 23:45:55 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:46:02 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/03 23:46:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:57160->10.242.8.142:8302: i/o timeout
2021/05/03 23:46:05 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:46:12 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.8.142
2021/05/03 23:46:12 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/03 23:46:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:57386->10.242.8.142:8302: i/o timeout
2021/05/03 23:46:15 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/03 23:46:20 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.8.142:8302
2021/05/03 23:47:12 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/03 23:48:18 [ERR] memberlist: failed to receive: read tcp 10.242.203.14:8302->10.242.8.142:40644: i/o timeout from=10.242.8.142:40644
2021/05/03 23:48:18 [ERR] memberlist: Failed to send error: write tcp 10.242.203.14:8302->10.242.8.142:40644: i/o timeout from=10.242.8.142:40644
2021/05/03 23:48:22 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/03 23:48:50 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.8.142:8302
2021/05/03 23:49:02 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/03 23:49:42 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/03 23:49:50 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.8.142:8302
2021/05/03 23:50:22 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/03 23:50:50 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.8.142:8302
2021/05/03 23:51:43 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/03 23:51:43 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.8.143:53924
2021/05/03 23:51:50 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.8.142:8302
2021/05/03 23:53:02 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/03 23:53:42 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/03 23:53:50 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.8.142:8302
2021/05/03 23:54:23 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/03 23:54:50 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.8.142:8302
2021/05/03 23:55:33 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/03 23:55:35 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/03 23:55:35 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:55:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:50724->10.242.8.142:8301: i/o timeout
2021/05/03 23:55:40 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/03 23:55:40 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2021/05/03 23:55:40 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/03 23:55:40 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/03 23:56:20 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.8.142:8302
2021/05/03 23:56:43 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/03 23:57:50 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.8.142:8302
2021/05/03 23:57:53 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/03 23:58:33 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/03 23:58:50 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.8.142:8302
2021/05/03 23:59:23 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/03 23:59:43 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/03 23:59:50 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.8.142:8302
2021/05/04 00:00:23 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/04 00:01:20 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.8.142:8302
2021/05/04 00:01:33 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/04 00:02:20 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.8.142:8302
2021/05/04 00:03:13 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/04 00:03:20 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.8.142:8302
2021/05/04 00:03:53 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/04 00:04:33 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/04 00:04:37 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/05/04 00:04:37 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:04:37 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:04:38 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.8.142
2021/05/04 00:04:38 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/04 00:04:39 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.8.142:60522
2021/05/04 00:04:39 [WARN] raft: Rejecting vote request from 10.242.8.142:8300 since we have a leader: 10.242.193.202:8300
2021/05/04 00:04:39 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:04:49 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:04:52 [INFO] serf: EventMemberUpdate: dev-consul-server-0.dc1
2021/05/04 00:04:52 [INFO] serf: EventMemberUpdate: dev-consul-server-0
2021/05/04 00:04:53 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:09:18 [ERR] yamux: keepalive failed: session shutdown
2021/05/04 00:09:17 [ERR] yamux: keepalive failed: session shutdown
2021/05/04 00:09:18 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-gbhmt)
2021/05/04 00:09:18 [ERR] yamux: keepalive failed: session shutdown
2021/05/04 00:09:18 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/04 00:09:17 [WARN] yamux: failed to send ping reply: session shutdown
2021/05/04 00:09:18 [ERR] yamux: keepalive failed: session shutdown
2021/05/04 00:09:18 [ERR] memberlist: Failed fallback ping: write tcp 10.242.203.14:59156->10.242.8.142:8302: i/o timeout
2021/05/04 00:09:18 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:09:18 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:35352
2021/05/04 00:09:18 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:36002
2021/05/04 00:09:18 [ERR] raft-net: Failed to flush response: write tcp 10.242.203.14:8300->10.242.193.202:55324: write: broken pipe
2021/05/04 00:09:18 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/04 00:09:18 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:35642
2021/05/04 00:09:18 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:35798
2021/05/04 00:09:18 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:35826
2021/05/04 00:09:18 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:35990
2021/05/04 00:09:18 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:35676
2021/05/04 00:09:18 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:35456
2021/05/04 00:09:18 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:35512
2021/05/04 00:09:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:44856->10.242.8.143:8301: i/o timeout
2021/05/04 00:09:27 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:10:34 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-gbhmt)
2021/05/04 00:10:42 [ERR] memberlist: Failed fallback ping: write tcp 10.242.203.14:48106->10.242.193.202:8301: i/o timeout
2021/05/04 00:10:42 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/04 00:10:42 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/05/04 00:10:45 [ERR] raft-net: Failed to flush response: write tcp 10.242.203.14:8300->10.242.193.202:36002: write: broken pipe
2021/05/04 00:10:46 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:38192
2021/05/04 00:10:46 [ERR] memberlist: Failed fallback ping: write tcp 10.242.203.14:46224->10.242.193.202:8302: i/o timeout
2021/05/04 00:10:46 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/04 00:11:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:47262->10.242.8.143:8301: i/o timeout
2021/05/04 00:11:49 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:11:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:47428->10.242.8.143:8301: i/o timeout
2021/05/04 00:11:59 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:12:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:47576->10.242.8.143:8301: i/o timeout
2021/05/04 00:12:07 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:12:11 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/05/04 00:12:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:47656->10.242.8.143:8301: i/o timeout
2021/05/04 00:12:12 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:12:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:42740->10.242.8.142:8301: i/o timeout
2021/05/04 00:12:17 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:12:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:47804->10.242.8.143:8301: i/o timeout
2021/05/04 00:12:20 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:12:21 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:12:21 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:12:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:42838->10.242.8.142:8301: i/o timeout
2021/05/04 00:12:22 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:12:26 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:12:26 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/04 00:12:26 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:12:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:42950->10.242.8.142:8301: i/o timeout
2021/05/04 00:12:27 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:12:32 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/05/04 00:12:32 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:12:32 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:12:34 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-gbhmt)
2021/05/04 00:12:34 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:12:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:48380->10.242.8.143:8301: i/o timeout
2021/05/04 00:12:48 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:12:49 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-gbhmt)
2021/05/04 00:13:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:48722->10.242.8.143:8301: i/o timeout
2021/05/04 00:15:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:51132->10.242.8.143:8301: i/o timeout
2021/05/04 00:15:27 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:15:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:51254->10.242.8.143:8301: i/o timeout
2021/05/04 00:15:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:15:38 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/05/04 00:15:38 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:15:38 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:15:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:46362->10.242.8.142:8301: i/o timeout
2021/05/04 00:15:42 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:15:44 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/04 00:15:44 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.8.143:39760
2021/05/04 00:15:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:46434->10.242.8.142:8301: i/o timeout
2021/05/04 00:15:45 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:15:46 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:15:46 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/04 00:15:46 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:15:48 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/05/04 00:15:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:38036->10.242.8.142:8302: i/o timeout
2021/05/04 00:15:54 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:15:54 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:15:55 [WARN] memberlist: Was able to connect to dev-consul-server-0.dc1 but other probes failed, network may be misconfigured
2021/05/04 00:15:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:46634->10.242.8.142:8301: i/o timeout
2021/05/04 00:15:56 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:16:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:46696->10.242.8.142:8301: i/o timeout
2021/05/04 00:16:00 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:16:02 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/04 00:16:02 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:16:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:46724->10.242.8.142:8301: i/o timeout
2021/05/04 00:16:03 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:16:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:38384->10.242.8.142:8302: i/o timeout
2021/05/04 00:16:10 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:16:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:38556->10.242.8.142:8302: i/o timeout
2021/05/04 00:16:20 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:16:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:38652->10.242.8.142:8302: i/o timeout
2021/05/04 00:16:30 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:16:33 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/04 00:16:37 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/04 00:16:38 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.8.142
2021/05/04 00:16:38 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/04 00:16:38 [ERR] memberlist: Failed fallback ping: write tcp 10.242.203.14:54532->10.242.193.202:8301: i/o timeout
2021/05/04 00:16:38 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/04 00:16:41 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-wjl58)
2021/05/04 00:16:50 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.8.142:8302
2021/05/04 00:17:18 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:44528
2021/05/04 00:17:18 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:44850
2021/05/04 00:17:18 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:44680
2021/05/04 00:17:18 [ERR] memberlist: Failed to send ack: write tcp 10.242.203.14:8301->10.242.193.202:47120: i/o timeout from=10.242.193.202:47120
2021/05/04 00:17:18 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/04 00:17:18 [ERR] memberlist: Push/Pull with dev-consul-7467bf7b46-wjl58 failed: EOF
2021/05/04 00:17:18 [ERR] memberlist: Failed fallback ping: write tcp 10.242.203.14:54816->10.242.193.202:8301: i/o timeout
2021/05/04 00:17:18 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/04 00:17:18 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/04 00:17:18 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:17:18 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:17:19 [ERR] raft-net: Failed to flush response: write tcp 10.242.203.14:8300->10.242.193.202:35990: write: broken pipe
2021/05/04 00:17:19 [WARN] raft: Failed to get previous log: 1958 log not found (last: 1956)
2021/05/04 00:17:19 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/04 00:17:19 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:17:19 [INFO] consul: New leader elected: dev-consul-server-1
2021/05/04 00:17:48 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/04 00:17:59 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:18:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:53716->10.242.8.143:8301: i/o timeout
2021/05/04 00:18:03 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:18:05 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:18:05 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.8.142:48378
2021/05/04 00:18:05 [WARN] raft: Rejecting vote request from 10.242.8.142:8300 since we have a leader: 10.242.193.202:8300
2021/05/04 00:18:07 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:18:07 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:18:07 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:18:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:53892->10.242.8.143:8301: i/o timeout
2021/05/04 00:18:13 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:18:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:48902->10.242.8.142:8301: i/o timeout
2021/05/04 00:18:17 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:18:17 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/04 00:18:17 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:18:17 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:18:20 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:18:20 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.8.142
2021/05/04 00:18:20 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:18:20 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/04 00:18:21 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/05/04 00:18:25 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-wjl58)
2021/05/04 00:18:34 [ERR] raft-net: Failed to flush response: write tcp 10.242.203.14:8300->10.242.193.202:44850: write: broken pipe
2021/05/04 00:18:36 [ERR] memberlist: Failed fallback ping: write tcp 10.242.203.14:49158->10.242.8.142:8301: i/o timeout
2021/05/04 00:18:36 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:18:44 [ERR] agent: failed to sync remote state: No cluster leader
2021/05/04 00:18:49 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:47398
2021/05/04 00:18:49 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:47330
2021/05/04 00:18:49 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:47406
2021/05/04 00:18:49 [ERR] memberlist: Failed fallback ping: write tcp 10.242.203.14:40914->10.242.8.142:8302: i/o timeout
2021/05/04 00:18:49 [ERR] memberlist: Failed fallback ping: write tcp 10.242.203.14:41800->10.242.95.21:8301: i/o timeout
2021/05/04 00:18:49 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:18:49 [ERR] agent: Coordinate update error: No cluster leader
2021/05/04 00:18:49 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/04 00:18:49 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wjl58 has failed, no acks received
2021/05/04 00:18:49 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:18:49 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:18:49 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-wjl58)
2021/05/04 00:18:49 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:18:49 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:18:49 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:18:49 [INFO] consul: New leader elected: dev-consul-server-1
2021/05/04 00:18:49 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:19:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:55624->10.242.8.143:8301: i/o timeout
2021/05/04 00:19:52 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:20:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:50810->10.242.8.142:8301: i/o timeout
2021/05/04 00:20:03 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:20:03 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:20:03 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/04 00:20:03 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:20:04 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/04 00:20:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:55856->10.242.8.143:8301: i/o timeout
2021/05/04 00:20:07 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:20:08 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:20:08 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:20:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:55938->10.242.8.143:8301: i/o timeout
2021/05/04 00:20:11 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:20:14 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:20:14 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/05/04 00:20:14 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:20:14 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:20:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:56094->10.242.8.143:8301: i/o timeout
2021/05/04 00:20:17 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:20:19 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:20:19 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:20:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:51170->10.242.8.142:8301: i/o timeout
2021/05/04 00:20:22 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:20:25 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/05/04 00:20:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:51298->10.242.8.142:8301: i/o timeout
2021/05/04 00:20:28 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:20:31 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/05/04 00:20:32 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/04 00:20:32 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:20:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:43036->10.242.8.142:8302: i/o timeout
2021/05/04 00:20:40 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:20:43 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:20:43 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:20:46 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:21:48 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/05/04 00:21:52 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-gbhmt)
2021/05/04 00:21:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:57932->10.242.8.143:8301: i/o timeout
2021/05/04 00:21:58 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:22:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:58050->10.242.8.143:8301: i/o timeout
2021/05/04 00:22:04 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:22:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:53070->10.242.8.142:8301: i/o timeout
2021/05/04 00:22:07 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:22:07 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:22:07 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/04 00:22:07 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:22:08 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:22:08 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:22:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:44648->10.242.8.142:8302: i/o timeout
2021/05/04 00:22:10 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:22:13 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/04 00:22:14 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:22:14 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:22:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:53282->10.242.8.142:8301: i/o timeout
2021/05/04 00:22:17 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:22:19 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/04 00:22:19 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:22:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:53308->10.242.8.142:8301: i/o timeout
2021/05/04 00:22:20 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:22:21 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/05/04 00:22:31 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/05/04 00:22:32 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:22:32 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:22:35 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:27:11 [ERR] yamux: keepalive failed: session shutdown
2021/05/04 00:27:11 [WARN] yamux: failed to send ping reply: session shutdown
2021/05/04 00:27:10 [WARN] yamux: failed to send ping reply: session shutdown
2021/05/04 00:27:11 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-gbhmt)
2021/05/04 00:27:10 [WARN] yamux: failed to send ping reply: session shutdown
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:53694
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:53534
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:54066
2021/05/04 00:27:11 [WARN] yamux: failed to send ping reply: session shutdown
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:54894
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:54544
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:54630
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:54724
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:54814
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:55040
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:55070
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:55320
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:55344
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:55496
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:55522
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:55526
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:55696
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:55826
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:55966
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:56074
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:56122
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:56150
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:53718
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:53856
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:53884
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:54034
2021/05/04 00:27:11 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:54242
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:54430
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:54378
2021/05/04 00:27:11 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:54198
2021/05/04 00:27:12 [INFO] consul: New leader elected: dev-consul-server-1
2021/05/04 00:27:12 [INFO] consul: New leader elected: dev-consul-server-1
2021/05/04 00:27:12 [WARN] raft: Rejecting vote request from 10.242.8.142:8300 since we have a leader: 10.242.193.202:8300
2021/05/04 00:27:13 [WARN] raft: Failed to get previous log: 2295 log not found (last: 2268)
2021/05/04 00:27:13 [WARN] raft: Failed to get previous log: 2298 log not found (last: 2268)
2021/05/04 00:27:13 [ERR] raft-net: Failed to flush response: write tcp 10.242.203.14:8300->10.242.193.202:47398: write: broken pipe
2021/05/04 00:28:47 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/04 00:29:04 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/04 00:29:04 [ERR] memberlist: Failed fallback ping: write tcp 10.242.203.14:58554->10.242.8.142:8301: i/o timeout
2021/05/04 00:29:04 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:32:43 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/05/04 00:32:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:39392->10.242.8.143:8301: i/o timeout
2021/05/04 00:32:44 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:32:46 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/05/04 00:32:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:34496->10.242.8.142:8301: i/o timeout
2021/05/04 00:32:51 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:32:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:39530->10.242.8.143:8301: i/o timeout
2021/05/04 00:32:54 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:32:54 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:32:54 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:32:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:39588->10.242.8.143:8301: i/o timeout
2021/05/04 00:32:57 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:32:57 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:33:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:39862->10.242.8.143:8301: i/o timeout
2021/05/04 00:33:11 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:33:11 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:33:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:34928->10.242.8.142:8301: i/o timeout
2021/05/04 00:33:16 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:33:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:54746->10.242.8.142:8302: i/o timeout
2021/05/04 00:33:20 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:33:20 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:33:20 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/04 00:33:20 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:33:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:35020->10.242.8.142:8301: i/o timeout
2021/05/04 00:33:21 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:33:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:54998->10.242.8.142:8302: i/o timeout
2021/05/04 00:33:35 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:33:43 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:33:43 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:33:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:35418->10.242.8.142:8301: i/o timeout
2021/05/04 00:33:44 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:33:45 [WARN] memberlist: Was able to connect to dev-consul-server-0.dc1 but other probes failed, network may be misconfigured
2021/05/04 00:33:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:35492->10.242.8.142:8301: i/o timeout
2021/05/04 00:33:48 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:33:48 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/04 00:33:48 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:33:48 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:33:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:55362->10.242.8.142:8302: i/o timeout
2021/05/04 00:33:55 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:34:04 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:34:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:55538->10.242.8.142:8302: i/o timeout
2021/05/04 00:34:05 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:34:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:40830->10.242.8.143:8301: i/o timeout
2021/05/04 00:34:06 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:34:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:40886->10.242.8.143:8301: i/o timeout
2021/05/04 00:34:09 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:34:10 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:34:10 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:34:11 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/04 00:34:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:55726->10.242.8.142:8302: i/o timeout
2021/05/04 00:34:15 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:34:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:55916->10.242.8.142:8302: i/o timeout
2021/05/04 00:34:25 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:34:25 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/04 00:34:25 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.8.142
2021/05/04 00:34:25 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/04 00:34:37 [ERR] memberlist: failed to receive: read tcp 10.242.203.14:8301->10.242.8.143:60036: i/o timeout from=10.242.8.143:60036
2021/05/04 00:34:37 [ERR] memberlist: Failed to send error: write tcp 10.242.203.14:8301->10.242.8.143:60036: i/o timeout from=10.242.8.143:60036
2021/05/04 00:34:37 [ERR] memberlist: failed to receive: read tcp 10.242.203.14:8301->10.242.8.142:39334: i/o timeout from=10.242.8.142:39334
2021/05/04 00:34:37 [ERR] memberlist: Failed to send error: write tcp 10.242.203.14:8301->10.242.8.142:39334: i/o timeout from=10.242.8.142:39334
2021/05/04 00:34:41 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.8.142:8302
2021/05/04 00:34:42 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/05/04 00:34:51 [ERR] memberlist: failed to receive: read tcp 10.242.203.14:8301->10.242.8.142:39400: i/o timeout from=10.242.8.142:39400
2021/05/04 00:34:51 [ERR] memberlist: Failed to send error: write tcp 10.242.203.14:8301->10.242.8.142:39400: i/o timeout from=10.242.8.142:39400
2021/05/04 00:34:52 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/04 00:39:50 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/04 00:39:50 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/04 00:39:50 [ERR] memberlist: Failed fallback ping: write tcp 10.242.203.14:44848->10.242.193.202:8301: i/o timeout
2021/05/04 00:39:50 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/04 00:39:50 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:39:50 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:39:50 [ERR] yamux: keepalive failed: session shutdown
2021/05/04 00:39:50 [WARN] yamux: failed to send ping reply: session shutdown
2021/05/04 00:39:50 [ERR] memberlist: Failed fallback ping: write tcp 10.242.203.14:42858->10.242.193.202:8302: i/o timeout
2021/05/04 00:39:50 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/04 00:39:50 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.8.142
2021/05/04 00:39:50 [WARN] yamux: failed to send ping reply: session shutdown
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:37296
2021/05/04 00:39:50 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/04 00:39:50 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:39:50 [WARN] yamux: failed to send ping reply: session shutdown
2021/05/04 00:39:50 [INFO] consul: New leader elected: dev-consul-server-1
2021/05/04 00:39:50 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/05/04 00:39:50 [INFO] consul: New leader elected: dev-consul-server-1
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:37466
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.8.142:38320
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:37146
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:38138
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:38302
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:38326
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:38472
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:38512
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:38124
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:38644
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:39012
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:38822
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:38896
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:39168
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:39078
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:39258
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:39344
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:39438
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:39522
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:39614
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:40060
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:40206
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:39722
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:39828
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:39918
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:41758
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:40416
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:40616
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:40778
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:40940
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:41112
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:41308
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:41498
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:37712
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:37866
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:37976
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:38712
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:37504
2021/05/04 00:39:50 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:37516
2021/05/04 00:39:51 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-wjl58)
2021/05/04 00:39:51 [WARN] raft: Rejecting vote request from 10.242.8.142:8300 since our last index is greater (2497, 2477)
2021/05/04 00:39:51 [WARN] raft: Rejecting vote request from 10.242.8.142:8300 since we have a leader: 10.242.193.202:8300
2021/05/04 00:39:51 [WARN] raft: Failed to get previous log: 2502 log not found (last: 2497)
2021/05/04 00:39:51 [WARN] raft: Failed to get previous log: 2502 log not found (last: 2497)
2021/05/04 00:39:51 [WARN] raft: Failed to get previous log: 2502 log not found (last: 2497)
2021/05/04 00:39:51 [WARN] raft: Failed to get previous log: 2502 log not found (last: 2497)
2021/05/04 00:39:51 [WARN] raft: Failed to get previous log: 2502 log not found (last: 2497)
2021/05/04 00:39:51 [WARN] raft: Failed to get previous log: 2502 log not found (last: 2497)
2021/05/04 00:39:51 [WARN] raft: Failed to get previous log: 2502 log not found (last: 2497)
2021/05/04 00:39:51 [WARN] raft: Failed to get previous log: 2502 log not found (last: 2497)
2021/05/04 00:39:51 [WARN] raft: Failed to get previous log: 2502 log not found (last: 2497)
2021/05/04 00:39:51 [WARN] raft: Failed to get previous log: 2502 log not found (last: 2497)
2021/05/04 00:39:51 [WARN] raft: Failed to get previous log: 2502 log not found (last: 2497)
2021/05/04 00:39:51 [WARN] raft: Failed to get previous log: 2502 log not found (last: 2497)
2021/05/04 00:39:51 [WARN] raft: Failed to get previous log: 2502 log not found (last: 2497)
2021/05/04 00:39:51 [WARN] raft: Failed to get previous log: 2502 log not found (last: 2497)
2021/05/04 00:39:51 [WARN] raft: Failed to get previous log: 2502 log not found (last: 2497)
2021/05/04 00:39:51 [WARN] raft: Failed to get previous log: 2502 log not found (last: 2497)
2021/05/04 00:39:51 [ERR] raft-net: Failed to flush response: write tcp 10.242.203.14:8300->10.242.193.202:56122: write: broken pipe
2021/05/04 00:39:53 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:42154
2021/05/04 00:40:01 [WARN] raft: Failed to get previous log: 2502 log not found (last: 2497)
2021/05/04 00:40:01 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:42628
2021/05/04 00:41:40 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:44890
2021/05/04 00:41:40 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:44870
2021/05/04 00:41:40 [WARN] raft: Heartbeat timeout from "10.242.193.202:8300" reached, starting election
2021/05/04 00:41:40 [INFO] raft: Node at 10.242.203.14:8300 [Candidate] entering Candidate state in term 28
2021/05/04 00:41:40 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/05/04 00:41:41 [ERR] memberlist: Failed fallback ping: write tcp 10.242.203.14:59358->10.242.8.142:8302: i/o timeout
2021/05/04 00:41:42 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:41:42 [ERR] memberlist: Failed fallback ping: write tcp 10.242.203.14:44630->10.242.8.143:8301: i/o timeout
2021/05/04 00:41:42 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:41:42 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/05/04 00:41:42 [ERR] raft-net: Failed to flush response: write tcp 10.242.203.14:8300->10.242.193.202:42154: write: broken pipe
2021/05/04 00:41:46 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-gbhmt)
2021/05/04 00:41:48 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.8.142:46904
2021/05/04 00:41:48 [INFO] raft: Duplicate RequestVote for same term: 28
2021/05/04 00:41:48 [WARN] raft: Failed to get previous log: 2677 log not found (last: 2672)
2021/05/04 00:41:48 [INFO] raft: Node at 10.242.203.14:8300 [Follower] entering Follower state (Leader: "10.242.8.142:8300")
2021/05/04 00:41:48 [INFO] consul: New leader elected: dev-consul-server-0
2021/05/04 00:41:49 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.8.142:46918
2021/05/04 00:42:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:46098->10.242.8.143:8301: i/o timeout
2021/05/04 00:42:59 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:43:07 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:43:13 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/05/04 00:43:13 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:43:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:46414->10.242.8.143:8301: i/o timeout
2021/05/04 00:43:15 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:43:18 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:43:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:46450->10.242.8.143:8301: i/o timeout
2021/05/04 00:43:19 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:43:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:41482->10.242.8.142:8301: i/o timeout
2021/05/04 00:43:22 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:43:22 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:43:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:46668->10.242.8.143:8301: i/o timeout
2021/05/04 00:43:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:41816->10.242.8.142:8301: i/o timeout
2021/05/04 00:43:38 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:43:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:46900->10.242.8.143:8301: i/o timeout
2021/05/04 00:43:42 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:43:42 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/05/04 00:43:42 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/04 00:43:42 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:43:43 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:43:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:33464->10.242.8.142:8302: i/o timeout
2021/05/04 00:43:45 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:43:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:41918->10.242.8.142:8301: i/o timeout
2021/05/04 00:43:45 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:43:45 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:43:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:47040->10.242.8.143:8301: i/o timeout
2021/05/04 00:43:49 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:43:50 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:43:50 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:43:51 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/04 00:43:52 [ERR] agent: Coordinate update error: No cluster leader
2021/05/04 00:43:52 [WARN] raft: Rejecting vote request from 10.242.193.202:8300 since we have a leader: 10.242.8.142:8300
2021/05/04 00:43:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:33650->10.242.8.142:8302: i/o timeout
2021/05/04 00:43:55 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:43:56 [ERR] raft-net: Failed to flush response: write tcp 10.242.203.14:8300->10.242.8.142:46904: write: connection reset by peer
2021/05/04 00:43:59 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/04 00:44:00 [WARN] raft: Rejecting vote request from 10.242.193.202:8300 since we have a leader: 10.242.8.142:8300
2021/05/04 00:44:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:33738->10.242.8.142:8302: i/o timeout
2021/05/04 00:44:07 [WARN] raft: Heartbeat timeout from "10.242.8.142:8300" reached, starting election
2021/05/04 00:44:07 [INFO] raft: Node at 10.242.203.14:8300 [Candidate] entering Candidate state in term 29
2021/05/04 00:44:07 [WARN] Unable to get address for server id b7a1a6f5-413c-0710-1713-92bc2833c11e, using fallback address 10.242.8.142:8300: Could not find address for server id b7a1a6f5-413c-0710-1713-92bc2833c11e
2021/05/04 00:44:07 [INFO] raft: Node at 10.242.203.14:8300 [Follower] entering Follower state (Leader: "")
2021/05/04 00:44:09 [WARN] raft: Rejecting vote request from 10.242.193.202:8300 since our last index is greater (2734, 2733)
2021/05/04 00:44:11 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:44:11 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:44:11 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:44:12 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/05/04 00:44:12 [INFO] raft: Node at 10.242.203.14:8300 [Candidate] entering Candidate state in term 32
2021/05/04 00:44:13 [INFO] raft: Election won. Tally: 2
2021/05/04 00:44:13 [INFO] raft: Node at 10.242.203.14:8300 [Leader] entering Leader state
2021/05/04 00:44:13 [INFO] raft: Added peer 3958e3ab-77f2-b5ab-abdd-054752b6ab35, starting replication
2021/05/04 00:44:13 [INFO] raft: Added peer b7a1a6f5-413c-0710-1713-92bc2833c11e, starting replication
2021/05/04 00:44:13 [INFO] consul: cluster leadership acquired
2021/05/04 00:44:13 [WARN] raft: AppendEntries to {Voter 3958e3ab-77f2-b5ab-abdd-054752b6ab35 10.242.193.202:8300} rejected, sending older logs (next: 2734)
2021/05/04 00:44:13 [INFO] consul: New leader elected: dev-consul-server-2
2021/05/04 00:44:13 [INFO] raft: pipelining replication to peer {Voter 3958e3ab-77f2-b5ab-abdd-054752b6ab35 10.242.193.202:8300}
2021/05/04 00:44:15 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-gbhmt but other probes failed, network may be misconfigured
2021/05/04 00:44:15 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: stream closed
2021/05/04 00:44:15 [WARN] raft: Failed to contact b7a1a6f5-413c-0710-1713-92bc2833c11e in 2.534968687s
2021/05/04 00:44:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:42508->10.242.8.142:8301: i/o timeout
2021/05/04 00:44:16 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:44:16 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/04 00:44:17 [ERR] raft: Failed to make RequestVote RPC to {Voter b7a1a6f5-413c-0710-1713-92bc2833c11e 10.242.8.142:8300}: read tcp 10.242.203.14:58096->10.242.8.142:8300: i/o timeout
2021/05/04 00:44:18 [WARN] raft: Failed to contact b7a1a6f5-413c-0710-1713-92bc2833c11e in 4.994002449s
2021/05/04 00:44:18 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/04 00:44:19 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/04 00:44:20 [INFO] raft: pipelining replication to peer {Voter b7a1a6f5-413c-0710-1713-92bc2833c11e 10.242.8.142:8300}
2021/05/04 00:44:21 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/04 00:44:24 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/04 00:44:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:42674->10.242.8.142:8301: i/o timeout
2021/05/04 00:44:25 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:44:25 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/04 00:44:25 [WARN] raft: Failed to contact b7a1a6f5-413c-0710-1713-92bc2833c11e in 2.500292447s
2021/05/04 00:44:27 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/04 00:44:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:47742->10.242.8.143:8301: i/o timeout
2021/05/04 00:44:28 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:44:28 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:44:28 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:44:28 [INFO] consul: member 'dev-consul-7467bf7b46-gbhmt' failed, marking health critical
2021/05/04 00:44:29 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/04 00:44:31 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/04 00:44:31 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:44:31 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/05/04 00:44:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:42736->10.242.8.142:8301: i/o timeout
2021/05/04 00:44:31 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:44:31 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/04 00:44:31 [WARN] raft: Failed to contact b7a1a6f5-413c-0710-1713-92bc2833c11e in 2.50019764s
2021/05/04 00:44:31 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/04 00:44:31 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:44:31 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:44:31 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/05/04 00:44:32 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:44:32 [INFO] consul: member 'dev-consul-7467bf7b46-gbhmt' joined, marking health alive
2021/05/04 00:44:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:47880->10.242.8.143:8301: i/o timeout
2021/05/04 00:44:36 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:44:38 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/04 00:44:38 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/04 00:44:38 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:44:38 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:44:38 [INFO] consul: member 'dev-consul-7467bf7b46-gbhmt' failed, marking health critical
2021/05/04 00:44:39 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/04 00:44:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:47926->10.242.8.143:8301: i/o timeout
2021/05/04 00:44:40 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:44:41 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:44:41 [INFO] consul: member 'dev-consul-7467bf7b46-gbhmt' joined, marking health alive
2021/05/04 00:44:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:43006->10.242.8.142:8301: i/o timeout
2021/05/04 00:44:44 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:44:44 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/04 00:44:44 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/04 00:44:45 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/04 00:44:45 [WARN] raft: Failed to contact b7a1a6f5-413c-0710-1713-92bc2833c11e in 2.500244897s
2021/05/04 00:44:47 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/04 00:44:48 [WARN] raft: Failed to contact b7a1a6f5-413c-0710-1713-92bc2833c11e in 4.979832573s
2021/05/04 00:44:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:48058->10.242.8.143:8301: i/o timeout
2021/05/04 00:44:48 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:44:48 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:44:48 [INFO] consul: member 'dev-consul-7467bf7b46-gbhmt' failed, marking health critical
2021/05/04 00:44:48 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/04 00:44:48 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:44:48 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/05/04 00:44:49 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/04 00:44:49 [WARN] Unable to get address for server id b7a1a6f5-413c-0710-1713-92bc2833c11e, using fallback address 10.242.8.142:8300: Could not find address for server id b7a1a6f5-413c-0710-1713-92bc2833c11e
2021/05/04 00:44:51 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/04 00:44:51 [WARN] raft: Failed to contact b7a1a6f5-413c-0710-1713-92bc2833c11e in 2.501357588s
2021/05/04 00:44:53 [ERR] raft: Failed to pipeline AppendEntries to {Voter b7a1a6f5-413c-0710-1713-92bc2833c11e 10.242.8.142:8300}: write tcp 10.242.203.14:60928->10.242.8.142:8300: use of closed network connection
2021/05/04 00:44:53 [INFO] raft: aborting pipeline replication to peer {Voter b7a1a6f5-413c-0710-1713-92bc2833c11e 10.242.8.142:8300}
2021/05/04 00:44:53 [WARN] Unable to get address for server id b7a1a6f5-413c-0710-1713-92bc2833c11e, using fallback address 10.242.8.142:8300: Could not find address for server id b7a1a6f5-413c-0710-1713-92bc2833c11e
2021/05/04 00:44:53 [WARN] Unable to get address for server id b7a1a6f5-413c-0710-1713-92bc2833c11e, using fallback address 10.242.8.142:8300: Could not find address for server id b7a1a6f5-413c-0710-1713-92bc2833c11e
2021/05/04 00:44:54 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:44:54 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:44:54 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/05/04 00:44:54 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/04 00:44:54 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/04 00:44:55 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/04 00:44:55 [WARN] raft: Failed to contact b7a1a6f5-413c-0710-1713-92bc2833c11e in 2.500248286s
2021/05/04 00:44:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:43224->10.242.8.142:8301: i/o timeout
2021/05/04 00:44:56 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:44:57 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/04 00:44:58 [WARN] raft: Failed to contact b7a1a6f5-413c-0710-1713-92bc2833c11e in 4.960615745s
2021/05/04 00:44:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:43256->10.242.8.142:8301: i/o timeout
2021/05/04 00:44:59 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:44:59 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/04 00:44:59 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:44:59 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/04 00:44:59 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:44:59 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/05/04 00:45:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:34806->10.242.8.142:8302: i/o timeout
2021/05/04 00:45:00 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:45:00 [WARN] raft: Failed to contact b7a1a6f5-413c-0710-1713-92bc2833c11e in 7.387584018s
2021/05/04 00:45:01 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/04 00:45:01 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/04 00:45:03 [ERR] raft: Failed to AppendEntries to {Voter b7a1a6f5-413c-0710-1713-92bc2833c11e 10.242.8.142:8300}: read tcp 10.242.203.14:60954->10.242.8.142:8300: i/o timeout
2021/05/04 00:45:03 [WARN] Unable to get address for server id b7a1a6f5-413c-0710-1713-92bc2833c11e, using fallback address 10.242.8.142:8300: Could not find address for server id b7a1a6f5-413c-0710-1713-92bc2833c11e
2021/05/04 00:45:03 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/04 00:45:03 [ERR] raft: Failed to heartbeat to 10.242.8.142:8300: read tcp 10.242.203.14:60926->10.242.8.142:8300: i/o timeout
2021/05/04 00:45:04 [WARN] Unable to get address for server id b7a1a6f5-413c-0710-1713-92bc2833c11e, using fallback address 10.242.8.142:8300: Could not find address for server id b7a1a6f5-413c-0710-1713-92bc2833c11e
2021/05/04 00:45:05 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/04 00:45:07 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/04 00:45:07 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:45:07 [INFO] consul: member 'dev-consul-7467bf7b46-gbhmt' joined, marking health alive
2021/05/04 00:45:07 [WARN] Unable to get address for server id b7a1a6f5-413c-0710-1713-92bc2833c11e, using fallback address 10.242.8.142:8300: Could not find address for server id b7a1a6f5-413c-0710-1713-92bc2833c11e
2021/05/04 00:45:07 [INFO] raft: pipelining replication to peer {Voter b7a1a6f5-413c-0710-1713-92bc2833c11e 10.242.8.142:8300}
2021/05/04 00:45:07 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:45:07 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:45:07 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/05/04 00:45:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:48810->10.242.8.143:8301: i/o timeout
2021/05/04 00:45:29 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:45:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:49020->10.242.8.143:8301: i/o timeout
2021/05/04 00:45:41 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:45:42 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/04 00:45:42 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/04 00:45:44 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:45:44 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:45:44 [INFO] consul: member 'dev-consul-7467bf7b46-gbhmt' failed, marking health critical
2021/05/04 00:45:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:49064->10.242.8.143:8301: i/o timeout
2021/05/04 00:45:45 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:45:46 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/04 00:45:46 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/04 00:45:47 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/04 00:45:49 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/05/04 00:45:50 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/04 00:45:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/04 00:45:51 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/04 00:45:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:44208->10.242.8.142:8301: i/o timeout
2021/05/04 00:45:52 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:45:53 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/04 00:45:53 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:45:53 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/05/04 00:45:54 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/04 00:45:54 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/04 00:45:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:35780->10.242.8.142:8302: i/o timeout
2021/05/04 00:45:55 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:45:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:44254->10.242.8.142:8301: i/o timeout
2021/05/04 00:45:55 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:45:55 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/04 00:45:55 [WARN] raft: Failed to contact b7a1a6f5-413c-0710-1713-92bc2833c11e in 2.500228862s
2021/05/04 00:45:56 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:45:56 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:45:56 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/05/04 00:45:58 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/04 00:45:59 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/04 00:46:04 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:46:04 [INFO] consul: member 'dev-consul-7467bf7b46-gbhmt' joined, marking health alive
2021/05/04 00:46:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:49476->10.242.8.143:8301: i/o timeout
2021/05/04 00:46:06 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:46:09 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:46:09 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:46:09 [INFO] consul: member 'dev-consul-7467bf7b46-gbhmt' failed, marking health critical
2021/05/04 00:46:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:49538->10.242.8.143:8301: i/o timeout
2021/05/04 00:46:10 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:46:10 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/04 00:46:10 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/04 00:46:11 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:46:11 [INFO] consul: member 'dev-consul-7467bf7b46-gbhmt' joined, marking health alive
2021/05/04 00:46:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:44686->10.242.8.142:8301: i/o timeout
2021/05/04 00:46:20 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:46:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:49852->10.242.8.143:8301: i/o timeout
2021/05/04 00:46:28 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:52:40 [ERR] yamux: keepalive failed: session shutdown
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:51010
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:51026
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:51168
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:51206
2021/05/04 00:52:41 [WARN] yamux: failed to send ping reply: session shutdown
2021/05/04 00:52:41 [ERR] yamux: Failed to write header: write tcp 10.242.203.14:8300->10.242.193.202:47368: use of closed network connection
2021/05/04 00:52:41 [ERR] raft: Failed to heartbeat to 10.242.193.202:8300: write tcp 10.242.203.14:60650->10.242.193.202:8300: i/o timeout
2021/05/04 00:52:40 [ERR] yamux: keepalive failed: session shutdown
2021/05/04 00:52:41 [ERR] yamux: keepalive failed: session shutdown
2021/05/04 00:52:41 [WARN] yamux: failed to send ping reply: session shutdown
2021/05/04 00:52:41 [ERR] yamux: keepalive failed: session shutdown
2021/05/04 00:52:40 [ERR] yamux: keepalive failed: session shutdown
2021/05/04 00:52:41 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-wjl58)
2021/05/04 00:52:41 [ERR] yamux: Failed to write header: write tcp 10.242.203.14:8300->10.242.95.21:60626: use of closed network connection
2021/05/04 00:52:41 [ERR] yamux: keepalive failed: session shutdown
2021/05/04 00:52:41 [ERR] yamux: keepalive failed: session shutdown
2021/05/04 00:52:40 [WARN] yamux: failed to send ping reply: session shutdown
2021/05/04 00:52:41 [ERR] raft: peer {Voter 3958e3ab-77f2-b5ab-abdd-054752b6ab35 10.242.193.202:8300} has newer term, stopping replication
2021/05/04 00:52:41 [INFO] raft: aborting pipeline replication to peer {Voter b7a1a6f5-413c-0710-1713-92bc2833c11e 10.242.8.142:8300}
2021/05/04 00:52:41 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:51342
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.8.142:60142
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:51680
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:54994
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:51560
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:51748
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:53352
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:52182
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.8.142:60152
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:53966
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:55474
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:55334
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:52676
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:53606
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:53784
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:54644
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:55822
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:52466
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:54788
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:52894
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:53008
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:53154
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:55686
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:51918
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:52540
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:52094
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:51384
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:51502
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:54304
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:55134
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:52018
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:52372
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:52758
2021/05/04 00:52:41 [INFO] raft: aborting pipeline replication to peer {Voter 3958e3ab-77f2-b5ab-abdd-054752b6ab35 10.242.193.202:8300}
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:52270
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:54126
2021/05/04 00:52:41 [WARN] yamux: failed to send ping reply: session shutdown
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:51848
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.8.142:60166
2021/05/04 00:52:41 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:54462
2021/05/04 00:52:41 [INFO] consul: New leader elected: dev-consul-server-1
2021/05/04 00:52:41 [INFO] raft: Node at 10.242.203.14:8300 [Follower] entering Follower state (Leader: "")
2021/05/04 00:52:41 [ERR] consul.rpc: multiplex conn accept failed: session shutdown from=10.242.193.202:52464
2021/05/04 00:52:41 [INFO] consul: New leader elected: dev-consul-server-0
2021/05/04 00:52:41 [ERR] consul: failed to wait for barrier: node is not the leader
2021/05/04 00:52:42 [WARN] raft: Rejecting vote request from 10.242.193.202:8300 since we have a leader: 10.242.8.142:8300
2021/05/04 00:52:42 [WARN] raft: Failed to get previous log: 2989 log not found (last: 2902)
2021/05/04 00:52:42 [INFO] consul: cluster leadership lost
2021/05/04 00:52:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:51848->10.242.8.143:8301: i/o timeout
2021/05/04 00:52:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:46912->10.242.8.142:8301: i/o timeout
2021/05/04 00:52:46 [WARN] consul.coordinate: Batch update failed: node is not the leader
2021/05/04 00:52:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:51954->10.242.8.143:8301: i/o timeout
2021/05/04 00:52:46 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:52:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:46982->10.242.8.142:8301: i/o timeout
2021/05/04 00:52:50 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:52:50 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:52:50 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/04 00:52:50 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:52:51 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:52:54 [WARN] raft: Rejecting vote request from 10.242.193.202:8300 since we have a leader: 10.242.8.142:8300
2021/05/04 00:52:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:38626->10.242.8.142:8302: i/o timeout
2021/05/04 00:52:55 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:53:02 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/05/04 00:53:02 [ERR] agent: failed to sync remote state: No cluster leader
2021/05/04 00:53:02 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/05/04 00:53:03 [WARN] raft: Rejecting vote request from 10.242.193.202:8300 since we have a leader: 10.242.8.142:8300
2021/05/04 00:53:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:38804->10.242.8.142:8302: i/o timeout
2021/05/04 00:53:05 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:53:09 [WARN] raft: Rejecting vote request from 10.242.193.202:8300 since we have a leader: 10.242.8.142:8300
2021/05/04 00:53:17 [WARN] raft: Rejecting vote request from 10.242.193.202:8300 since we have a leader: 10.242.8.142:8300
2021/05/04 00:53:18 [ERR] agent: Coordinate update error: No cluster leader
2021/05/04 00:53:18 [WARN] raft: Heartbeat timeout from "10.242.8.142:8300" reached, starting election
2021/05/04 00:53:18 [INFO] raft: Node at 10.242.203.14:8300 [Candidate] entering Candidate state in term 36
2021/05/04 00:53:19 [WARN] Unable to get address for server id b7a1a6f5-413c-0710-1713-92bc2833c11e, using fallback address 10.242.8.142:8300: Could not find address for server id b7a1a6f5-413c-0710-1713-92bc2833c11e
2021/05/04 00:53:19 [INFO] raft: Node at 10.242.203.14:8300 [Follower] entering Follower state (Leader: "")
2021/05/04 00:53:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:39046->10.242.8.142:8302: i/o timeout
2021/05/04 00:53:20 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:53:21 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/04 00:53:21 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.8.142:60168
2021/05/04 00:53:22 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/04 00:53:22 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/04 00:53:22 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.8.142
2021/05/04 00:53:22 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/04 00:53:22 [INFO] consul: New leader elected: dev-consul-server-1
2021/05/04 00:53:23 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:57878
2021/05/04 00:53:29 [ERR] raft: Failed to make RequestVote RPC to {Voter b7a1a6f5-413c-0710-1713-92bc2833c11e 10.242.8.142:8300}: read tcp 10.242.203.14:33624->10.242.8.142:8300: i/o timeout
2021/05/04 00:53:33 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:53:33 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:53:33 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:53:35 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/05/04 00:53:38 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:53:38 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:53:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:52934->10.242.8.143:8301: i/o timeout
2021/05/04 00:53:39 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:53:41 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.8.142
2021/05/04 00:53:41 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/04 00:53:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:47956->10.242.8.142:8301: i/o timeout
2021/05/04 00:53:41 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:53:41 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:53:41 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/04 00:53:41 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:53:43 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:53:43 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:53:43 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:54:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:53500->10.242.8.143:8301: i/o timeout
2021/05/04 00:54:05 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:54:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:53540->10.242.8.143:8301: i/o timeout
2021/05/04 00:54:08 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:54:11 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:54:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:48574->10.242.8.142:8301: i/o timeout
2021/05/04 00:54:14 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:54:17 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-wjl58)
2021/05/04 00:54:17 [ERR] memberlist: Failed fallback ping: write tcp 10.242.203.14:48672->10.242.8.142:8301: i/o timeout
2021/05/04 00:54:17 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:54:22 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:54:22 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/04 00:54:22 [ERR] raft-net: Failed to flush response: write tcp 10.242.203.14:8300->10.242.193.202:55822: write: broken pipe
2021/05/04 00:54:22 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:54:22 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/04 00:54:25 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/05/04 00:54:25 [ERR] memberlist: Failed fallback ping: write tcp 10.242.203.14:55972->10.242.193.202:8301: i/o timeout
2021/05/04 00:54:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:40352->10.242.8.142:8302: i/o timeout
2021/05/04 00:54:27 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:54:30 [INFO] consul: New leader elected: dev-consul-server-1
2021/05/04 00:54:31 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:59388
2021/05/04 00:54:40 [WARN] memberlist: Was able to connect to dev-consul-server-0.dc1 but other probes failed, network may be misconfigured
2021/05/04 00:54:40 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/04 00:54:41 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:54:41 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:54:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:49190->10.242.8.142:8301: i/o timeout
2021/05/04 00:54:44 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:54:44 [ERR] memberlist: failed to receive: read tcp 10.242.203.14:8301->10.242.8.143:55996: i/o timeout from=10.242.8.143:55996
2021/05/04 00:54:44 [ERR] memberlist: Failed to send error: write tcp 10.242.203.14:8301->10.242.8.143:55996: i/o timeout from=10.242.8.143:55996
2021/05/04 00:54:46 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/04 00:54:46 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:54:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:49238->10.242.8.142:8301: i/o timeout
2021/05/04 00:54:47 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:54:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:40828->10.242.8.142:8302: i/o timeout
2021/05/04 00:54:50 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:54:52 [ERR] memberlist: Push/Pull with dev-consul-server-0 failed: read tcp 10.242.203.14:49160->10.242.8.142:8301: i/o timeout
2021/05/04 00:55:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:41002->10.242.8.142:8302: i/o timeout
2021/05/04 00:55:00 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:55:05 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:55:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:54632->10.242.8.143:8301: i/o timeout
2021/05/04 00:55:07 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:55:07 [WARN] raft: Rejecting vote request from 10.242.8.142:8300 since we have a leader: 10.242.193.202:8300
2021/05/04 00:55:07 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:55:07 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:55:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:49644->10.242.8.142:8301: i/o timeout
2021/05/04 00:55:10 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:55:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:49700->10.242.8.142:8301: i/o timeout
2021/05/04 00:55:12 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:55:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:54786->10.242.8.143:8301: i/o timeout
2021/05/04 00:55:15 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:55:17 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:55:17 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:55:17 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:55:17 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/04 00:55:17 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:55:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:49782->10.242.8.142:8301: i/o timeout
2021/05/04 00:55:18 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:55:20 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.8.142:8301
2021/05/04 00:55:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:41460->10.242.8.142:8302: i/o timeout
2021/05/04 00:55:25 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:55:28 [ERR] memberlist: failed to receive: read tcp 10.242.203.14:8301->10.242.8.143:56572: i/o timeout from=10.242.8.143:56572
2021/05/04 00:55:28 [ERR] memberlist: Failed to send error: write tcp 10.242.203.14:8301->10.242.8.143:56572: i/o timeout from=10.242.8.143:56572
2021/05/04 00:55:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:41620->10.242.8.142:8302: i/o timeout
2021/05/04 00:55:35 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:55:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:41708->10.242.8.142:8302: i/o timeout
2021/05/04 00:55:45 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:55:51 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/04 00:55:51 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.8.143:58372
2021/05/04 00:55:55 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/04 00:55:55 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.8.142
2021/05/04 00:55:55 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/04 00:55:56 [ERR] memberlist: failed to receive: read tcp 10.242.203.14:8302->10.242.8.142:45322: i/o timeout from=10.242.8.142:45322
2021/05/04 00:55:56 [ERR] memberlist: Failed to send error: write tcp 10.242.203.14:8302->10.242.8.142:45322: i/o timeout from=10.242.8.142:45322
2021/05/04 00:56:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:42094->10.242.8.142:8302: i/o timeout
2021/05/04 00:56:00 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:56:00 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/04 00:56:11 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.8.142:8302
2021/05/04 00:56:12 [ERR] memberlist: failed to receive: read tcp 10.242.203.14:8301->10.242.8.142:36328: i/o timeout from=10.242.8.142:36328
2021/05/04 00:56:12 [ERR] memberlist: Failed to send error: write tcp 10.242.203.14:8301->10.242.8.142:36328: i/o timeout from=10.242.8.142:36328
2021/05/04 00:56:13 [ERR] memberlist: failed to receive: read tcp 10.242.203.14:8301->10.242.8.143:57050: i/o timeout from=10.242.8.143:57050
2021/05/04 00:56:13 [ERR] memberlist: Failed to send error: write tcp 10.242.203.14:8301->10.242.8.143:57050: i/o timeout from=10.242.8.143:57050
2021/05/04 00:56:29 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.8.142
2021/05/04 00:56:29 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/04 00:56:31 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:56:31 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:56:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:51180->10.242.8.142:8301: i/o timeout
2021/05/04 00:56:33 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:56:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:42728->10.242.8.142:8302: i/o timeout
2021/05/04 00:56:35 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:56:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:51226->10.242.8.142:8301: i/o timeout
2021/05/04 00:56:36 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/05/04 00:56:37 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.8.142
2021/05/04 00:56:37 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:56:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:42830->10.242.8.142:8302: i/o timeout
2021/05/04 00:56:45 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:56:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:43134->10.242.8.142:8302: i/o timeout
2021/05/04 00:56:55 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:56:55 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:56:55 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/05/04 00:56:57 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-gbhmt but other probes failed, network may be misconfigured
2021/05/04 00:57:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:56730->10.242.8.143:8301: i/o timeout
2021/05/04 00:57:00 [INFO] memberlist: Suspect dev-consul-7467bf7b46-gbhmt has failed, no acks received
2021/05/04 00:57:01 [INFO] memberlist: Marking dev-consul-7467bf7b46-gbhmt as failed, suspect timeout reached (2 peer confirmations)
2021/05/04 00:57:01 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:57:03 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/05/04 00:57:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:43520->10.242.8.142:8302: i/o timeout
2021/05/04 00:57:15 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:57:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:43714->10.242.8.142:8302: i/o timeout
2021/05/04 00:57:25 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:57:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.203.14:43984->10.242.8.142:8302: i/o timeout
2021/05/04 00:57:40 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/05/04 00:57:40 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/04 00:57:42 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/04 00:57:42 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.8.142
2021/05/04 00:57:42 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/04 00:58:20 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-gbhmt 10.242.8.143:8301
2021/05/04 00:58:29 [ERR] memberlist: failed to receive: read tcp 10.242.203.14:8301->10.242.8.143:57850: i/o timeout from=10.242.8.143:57850
2021/05/04 00:58:29 [ERR] memberlist: Failed to send error: write tcp 10.242.203.14:8301->10.242.8.143:57850: i/o timeout from=10.242.8.143:57850
2021/05/04 00:58:32 [WARN] raft: Rejecting vote request from 10.242.8.142:8300 since we have a leader: 10.242.193.202:8300
2021/05/04 00:58:36 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.8.142
2021/05/04 00:58:36 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/04 00:58:36 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.8.142
2021/05/04 00:58:36 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.8.142:8300) (DC: dc1)
2021/05/04 00:58:36 [WARN] memberlist: Refuting a dead message (from: dev-consul-server-0)
2021/05/04 00:58:36 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-gbhmt 10.242.8.143
2021/05/04 00:58:41 [WARN] raft: Rejecting vote request from 10.242.8.142:8300 since we have a leader: 10.242.193.202:8300
2021/05/04 00:58:47 [INFO] consul: New leader elected: dev-consul-server-1
2021/05/04 00:58:48 [DEBUG] raft-net: 10.242.203.14:8300 accepted connection from: 10.242.193.202:37092