By type
+ CONSUL_BIND=
+ [ -n ]
+ CONSUL_CLIENT=
+ [ -n ]
+ CONSUL_DATA_DIR=/consul/data
+ CONSUL_CONFIG_DIR=/consul/config
+ [ -n ]
+ [ a = - ]
+ [ agent = agent ]
+ shift
+ set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
+ [ consul = consul ]
+ stat -c %u /consul/data
+ id -u consul
+ [ 100 != 100 ]
+ stat -c %u /consul/config
+ id -u consul
+ [ 100 != 100 ]
+ [ ! -z ]
+ exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
bootstrap_expect > 0: expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
Version: 'v1.0.6'
Node ID: 'dff60f1e-22b9-7f6a-153b-e2e4aea758bb'
Node name: 'dev-consul-server-0'
Datacenter: 'dc1' (Segment: '')
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.242.113.198 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2021/05/07 10:43:00 [INFO] raft: Initial configuration (index=0): []
2021/05/07 10:43:00 [INFO] raft: Node at 10.242.113.198:8300 [Follower] entering Follower state (Leader: "")
2021/05/07 10:43:00 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.113.198
2021/05/07 10:43:00 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.113.198
2021/05/07 10:43:00 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/07 10:43:00 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.113.198:8300) (DC: dc1)
2021/05/07 10:43:00 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2021/05/07 10:43:00 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2021/05/07 10:43:00 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
2021/05/07 10:43:00 [INFO] agent: started state syncer
2021/05/07 10:43:00 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer
2021/05/07 10:43:00 [INFO] agent: Joining LAN cluster...
2021/05/07 10:43:00 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/05/07 10:43:01 [WARN] memberlist: Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
2021/05/07 10:43:01 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
2021/05/07 10:43:02 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/05/07 10:43:02 [INFO] agent: (LAN) joined: 0 Err: 3 error(s) occurred:
* Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/05/07 10:43:02 [WARN] agent: Join LAN failed: , retrying in 30s
2021/05/07 10:43:08 [ERR] agent: failed to sync remote state: No cluster leader
2021/05/07 10:43:10 [WARN] raft: no known peers, aborting election
2021/05/07 10:43:32 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/05/07 10:43:32 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
2021/05/07 10:43:32 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/05/07 10:43:32 [INFO] agent: (LAN) joined: 1 Err:
2021/05/07 10:43:32 [INFO] agent: Join LAN completed. Synced with 1 initial agents
==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=114b66ef-bf1d-9d30-669e-82d213fcacdb&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2021/05/07 10:43:37 [ERR] agent: Coordinate update error: No cluster leader
2021/05/07 10:43:40 [ERR] agent: failed to sync remote state: No cluster leader
2021/05/07 10:43:41 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 10:43:41 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 10:43:41 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 10:43:41 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 10:43:41 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 10:43:54 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.139.204
2021/05/07 10:43:54 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.139.204:8300) (DC: dc1)
2021/05/07 10:43:54 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.242.139.204:8300,10.242.113.198:8300,10.242.41.146:8300
2021/05/07 10:43:54 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.139.204
2021/05/07 10:43:54 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/05/07 10:44:00 [DEBUG] raft-net: 10.242.113.198:8300 accepted connection from: 10.242.139.204:35338
2021/05/07 10:44:00 [DEBUG] raft-net: 10.242.113.198:8300 accepted connection from: 10.242.139.204:35340
2021/05/07 10:44:00 [INFO] consul: New leader elected: dev-consul-server-2
2021/05/07 10:44:01 [DEBUG] raft-net: 10.242.113.198:8300 accepted connection from: 10.242.139.204:35346
2021/05/07 10:44:04 [INFO] agent: Synced node info
2021/05/07 11:05:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36254->10.242.41.137:8301: i/o timeout
2021/05/07 11:05:56 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:06:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36324->10.242.41.137:8301: i/o timeout
2021/05/07 11:06:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36350->10.242.41.137:8301: i/o timeout
2021/05/07 11:06:02 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:06:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:60748->10.242.41.146:8301: i/o timeout
2021/05/07 11:06:30 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:06:31 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/07 11:06:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36730->10.242.41.137:8301: i/o timeout
2021/05/07 11:06:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:07:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:37512->10.242.41.137:8301: i/o timeout
2021/05/07 11:07:40 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:07:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:37530->10.242.41.137:8301: i/o timeout
2021/05/07 11:07:42 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:07:45 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/07 11:07:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:37600->10.242.41.137:8301: i/o timeout
2021/05/07 11:07:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:33472->10.242.41.146:8301: i/o timeout
2021/05/07 11:07:52 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:08:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:33562->10.242.41.146:8301: i/o timeout
2021/05/07 11:08:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:37760->10.242.41.137:8301: i/o timeout
2021/05/07 11:08:02 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:08:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:33588->10.242.41.146:8301: i/o timeout
2021/05/07 11:08:05 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:08:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:37794->10.242.41.137:8301: i/o timeout
2021/05/07 11:08:07 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:08:08 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-x5pmr)
2021/05/07 11:08:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:37856->10.242.41.137:8301: i/o timeout
2021/05/07 11:08:11 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:08:12 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:08:13 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:08:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:37888->10.242.41.137:8301: i/o timeout
2021/05/07 11:08:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:33848->10.242.41.146:8301: i/o timeout
2021/05/07 11:08:28 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:08:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:38058->10.242.41.137:8301: i/o timeout
2021/05/07 11:08:31 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:08:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:38074->10.242.41.137:8301: i/o timeout
2021/05/07 11:08:33 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:08:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:33914->10.242.41.146:8301: i/o timeout
2021/05/07 11:08:35 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:08:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:38110->10.242.41.137:8301: i/o timeout
2021/05/07 11:08:37 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:08:39 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:08:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:33994->10.242.41.146:8301: i/o timeout
2021/05/07 11:08:40 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:08:46 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:08:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:34160->10.242.41.146:8301: i/o timeout
2021/05/07 11:08:53 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:08:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:38398->10.242.41.137:8301: i/o timeout
2021/05/07 11:08:58 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:09:00 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/07 11:10:42 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-x5pmr but other probes failed, network may be misconfigured
2021/05/07 11:10:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:39736->10.242.41.137:8301: i/o timeout
2021/05/07 11:10:43 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:10:45 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-x5pmr)
2021/05/07 11:10:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:35602->10.242.41.146:8301: i/o timeout
2021/05/07 11:10:47 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:10:48 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:10:48 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:10:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:42622->10.242.41.146:8302: i/o timeout
2021/05/07 11:10:50 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:10:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:39800->10.242.41.137:8301: i/o timeout
2021/05/07 11:10:50 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:10:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:39838->10.242.41.137:8301: i/o timeout
2021/05/07 11:10:53 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:10:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:39882->10.242.41.137:8301: i/o timeout
2021/05/07 11:10:56 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:10:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:39904->10.242.41.137:8301: i/o timeout
2021/05/07 11:10:58 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:11:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:39942->10.242.41.137:8301: i/o timeout
2021/05/07 11:11:00 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:11:00 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:11:02 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:11:02 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:11:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:39956->10.242.41.137:8301: i/o timeout
2021/05/07 11:11:02 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:11:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:42778->10.242.41.146:8302: i/o timeout
2021/05/07 11:11:05 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:11:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:42888->10.242.41.146:8302: i/o timeout
2021/05/07 11:11:15 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:11:16 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:11:16 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:11:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:35940->10.242.41.146:8301: i/o timeout
2021/05/07 11:11:17 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:11:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:35968->10.242.41.146:8301: i/o timeout
2021/05/07 11:11:19 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:11:20 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:11:20 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:11:20 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:11:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36002->10.242.41.146:8301: i/o timeout
2021/05/07 11:11:22 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:11:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36012->10.242.41.146:8301: i/o timeout
2021/05/07 11:11:24 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:11:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43014->10.242.41.146:8302: i/o timeout
2021/05/07 11:11:25 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:11:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36076->10.242.41.146:8301: i/o timeout
2021/05/07 11:11:27 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:11:29 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:11:29 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:11:29 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:11:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36110->10.242.41.146:8301: i/o timeout
2021/05/07 11:11:29 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:11:41 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:11:56 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8301->10.242.41.146:34284: i/o timeout from=10.242.41.146:34284
2021/05/07 11:11:56 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8301->10.242.41.146:34284: i/o timeout from=10.242.41.146:34284
2021/05/07 11:12:05 [DEBUG] raft-net: 10.242.113.198:8300 accepted connection from: 10.242.41.146:60542
2021/05/07 11:12:05 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 11:12:21 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:12:33 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:12:33 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:57612
2021/05/07 11:13:00 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:13:01 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:13:41 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:14:21 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:14:50 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:14:52 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:14:52 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:14:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:42730->10.242.41.137:8301: i/o timeout
2021/05/07 11:14:53 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:14:55 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:14:55 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:14:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:42754->10.242.41.137:8301: i/o timeout
2021/05/07 11:14:55 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:15:01 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:15:10 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 11:15:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:45844->10.242.41.146:8302: i/o timeout
2021/05/07 11:15:20 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:15:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:45968->10.242.41.146:8302: i/o timeout
2021/05/07 11:15:30 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:15:35 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:15:35 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:15:41 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:15:41 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:15:43 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 11:15:43 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/07 11:15:43 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:15:43 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:15:43 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:15:43 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:15:43 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:15:43 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-x5pmr but other probes failed, network may be misconfigured
2021/05/07 11:15:44 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:15:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:39192->10.242.41.146:8301: i/o timeout
2021/05/07 11:15:47 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:15:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43390->10.242.41.137:8301: i/o timeout
2021/05/07 11:15:50 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:15:59 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-x5pmr but other probes failed, network may be misconfigured
2021/05/07 11:16:00 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:16:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43566->10.242.41.137:8301: i/o timeout
2021/05/07 11:16:03 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:16:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43590->10.242.41.137:8301: i/o timeout
2021/05/07 11:16:05 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:16:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43634->10.242.41.137:8301: i/o timeout
2021/05/07 11:16:08 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:16:09 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:16:09 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:16:10 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:16:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43704->10.242.41.137:8301: i/o timeout
2021/05/07 11:16:14 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:16:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43760->10.242.41.137:8301: i/o timeout
2021/05/07 11:16:17 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:16:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:39612->10.242.41.146:8301: i/o timeout
2021/05/07 11:16:20 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:16:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:39770->10.242.41.146:8301: i/o timeout
2021/05/07 11:16:34 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:16:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46764->10.242.41.146:8302: i/o timeout
2021/05/07 11:16:35 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:16:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43986->10.242.41.137:8301: i/o timeout
2021/05/07 11:16:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:44010->10.242.41.137:8301: i/o timeout
2021/05/07 11:16:38 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:16:41 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:16:41 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:16:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:44052->10.242.41.137:8301: i/o timeout
2021/05/07 11:16:41 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:16:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:39880->10.242.41.146:8301: i/o timeout
2021/05/07 11:16:44 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:16:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:39914->10.242.41.146:8301: i/o timeout
2021/05/07 11:16:48 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:16:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:39994->10.242.41.146:8301: i/o timeout
2021/05/07 11:16:53 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:16:55 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/07 11:16:56 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 11:16:57 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 11:16:57 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:16:58 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:17:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:44414->10.242.41.137:8301: i/o timeout
2021/05/07 11:17:11 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:17:13 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:17:13 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:17:14 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:17:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:40466->10.242.41.146:8301: i/o timeout
2021/05/07 11:17:30 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:17:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:44686->10.242.41.137:8301: i/o timeout
2021/05/07 11:17:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:17:35 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:17:35 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:17:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:44702->10.242.41.137:8301: i/o timeout
2021/05/07 11:17:37 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:17:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47568->10.242.41.146:8302: i/o timeout
2021/05/07 11:17:40 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:17:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:40578->10.242.41.146:8301: i/o timeout
2021/05/07 11:17:41 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:17:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:40614->10.242.41.146:8301: i/o timeout
2021/05/07 11:17:47 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:17:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:40730->10.242.41.146:8301: i/o timeout
2021/05/07 11:17:51 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:17:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:40742->10.242.41.146:8301: i/o timeout
2021/05/07 11:17:53 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:17:54 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:17:54 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:58776
2021/05/07 11:17:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:40772->10.242.41.146:8301: i/o timeout
2021/05/07 11:17:55 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:17:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:40814->10.242.41.146:8301: i/o timeout
2021/05/07 11:17:57 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:17:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:40838->10.242.41.146:8301: i/o timeout
2021/05/07 11:17:59 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:18:00 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:18:00 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:18:00 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:18:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47828->10.242.41.146:8302: i/o timeout
2021/05/07 11:18:00 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:18:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47948->10.242.41.146:8302: i/o timeout
2021/05/07 11:18:10 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:18:13 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:18:20 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 11:18:23 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:18:23 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:18:24 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:18:24 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:18:24 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:18:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:41146->10.242.41.146:8301: i/o timeout
2021/05/07 11:18:25 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:18:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:41160->10.242.41.146:8301: i/o timeout
2021/05/07 11:18:27 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:18:28 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:18:28 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:18:28 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:18:30 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:18:49 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:19:23 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:19:23 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:19:24 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:19:24 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:19:24 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 11:19:24 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:19:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:41864->10.242.41.146:8301: i/o timeout
2021/05/07 11:19:25 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:19:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:41902->10.242.41.146:8301: i/o timeout
2021/05/07 11:19:27 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:19:28 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:19:28 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:19:28 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:19:29 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:19:29 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:19:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46142->10.242.41.137:8301: i/o timeout
2021/05/07 11:19:32 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:19:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46154->10.242.41.137:8301: i/o timeout
2021/05/07 11:19:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:19:35 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:19:35 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:19:35 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 11:19:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46186->10.242.41.137:8301: i/o timeout
2021/05/07 11:19:36 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:19:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:42032->10.242.41.146:8301: i/o timeout
2021/05/07 11:19:39 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:19:41 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8301->10.242.41.146:38794: i/o timeout from=10.242.41.146:38794
2021/05/07 11:19:41 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8301->10.242.41.146:38794: i/o timeout from=10.242.41.146:38794
2021/05/07 11:19:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46246->10.242.41.137:8301: i/o timeout
2021/05/07 11:19:41 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:19:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:42094->10.242.41.146:8301: i/o timeout
2021/05/07 11:19:43 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:19:44 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/07 11:19:45 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 11:19:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46308->10.242.41.137:8301: i/o timeout
2021/05/07 11:19:46 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:19:48 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:19:48 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:19:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:42160->10.242.41.146:8301: i/o timeout
2021/05/07 11:19:48 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:19:50 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:19:50 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:19:50 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:19:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:42196->10.242.41.146:8301: i/o timeout
2021/05/07 11:19:50 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:19:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:49226->10.242.41.146:8302: i/o timeout
2021/05/07 11:19:55 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:20:00 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:20:00 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:20:00 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:20:01 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:20:19 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:20:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46742->10.242.41.137:8301: i/o timeout
2021/05/07 11:20:20 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:20:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46772->10.242.41.137:8301: i/o timeout
2021/05/07 11:20:23 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:20:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46796->10.242.41.137:8301: i/o timeout
2021/05/07 11:20:25 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:20:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46840->10.242.41.137:8301: i/o timeout
2021/05/07 11:20:28 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:20:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46872->10.242.41.137:8301: i/o timeout
2021/05/07 11:20:30 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:20:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46900->10.242.41.137:8301: i/o timeout
2021/05/07 11:20:33 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:20:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46922->10.242.41.137:8301: i/o timeout
2021/05/07 11:20:35 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:20:37 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:20:37 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:20:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46942->10.242.41.137:8301: i/o timeout
2021/05/07 11:20:37 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:20:40 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:20:40 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:20:41 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:20:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:49840->10.242.41.146:8302: i/o timeout
2021/05/07 11:20:45 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:20:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:49962->10.242.41.146:8302: i/o timeout
2021/05/07 11:20:55 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:21:03 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:21:03 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:33826
2021/05/07 11:21:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:50076->10.242.41.146:8302: i/o timeout
2021/05/07 11:21:05 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:21:07 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 11:21:07 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:21:07 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:21:08 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:21:09 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:21:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43196->10.242.41.146:8301: i/o timeout
2021/05/07 11:21:12 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:21:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43208->10.242.41.146:8301: i/o timeout
2021/05/07 11:21:14 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:21:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43266->10.242.41.146:8301: i/o timeout
2021/05/07 11:21:17 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:21:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43304->10.242.41.146:8301: i/o timeout
2021/05/07 11:21:20 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:21:21 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:21:22 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:21:23 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:21:23 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:21:29 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:21:29 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-x5pmr)
2021/05/07 11:21:30 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 11:21:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47626->10.242.41.137:8301: i/o timeout
2021/05/07 11:21:32 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:21:34 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-x5pmr but other probes failed, network may be misconfigured
2021/05/07 11:21:36 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:21:36 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:21:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47672->10.242.41.137:8301: i/o timeout
2021/05/07 11:21:36 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:21:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47702->10.242.41.137:8301: i/o timeout
2021/05/07 11:21:38 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:21:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43554->10.242.41.146:8301: i/o timeout
2021/05/07 11:21:41 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:21:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43604->10.242.41.146:8301: i/o timeout
2021/05/07 11:21:45 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:21:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47808->10.242.41.137:8301: i/o timeout
2021/05/07 11:21:48 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:21:51 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:21:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47902->10.242.41.137:8301: i/o timeout
2021/05/07 11:21:53 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:21:56 [ERR] memberlist: Push/Pull with dev-consul-7467bf7b46-x5pmr failed: read tcp 10.242.113.198:47814->10.242.41.137:8301: i/o timeout
2021/05/07 11:21:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47908->10.242.41.137:8301: i/o timeout
2021/05/07 11:21:56 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:22:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47992->10.242.41.137:8301: i/o timeout
2021/05/07 11:22:01 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:22:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43840->10.242.41.146:8301: i/o timeout
2021/05/07 11:22:04 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:22:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:48058->10.242.41.137:8301: i/o timeout
2021/05/07 11:22:06 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:22:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:48086->10.242.41.137:8301: i/o timeout
2021/05/07 11:22:08 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:22:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43924->10.242.41.146:8301: i/o timeout
2021/05/07 11:22:10 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:22:13 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:22:13 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:22:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43966->10.242.41.146:8301: i/o timeout
2021/05/07 11:22:13 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:22:14 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:22:14 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:35458
2021/05/07 11:22:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:50968->10.242.41.146:8302: i/o timeout
2021/05/07 11:22:15 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:22:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43988->10.242.41.146:8301: i/o timeout
2021/05/07 11:22:15 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:22:16 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:22:16 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:22:16 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:22:17 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:22:18 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 11:22:18 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:22:18 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:22:21 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:22:24 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 11:22:26 [WARN] raft: Heartbeat timeout from "10.242.139.204:8300" reached, starting election
2021/05/07 11:22:26 [INFO] raft: Node at 10.242.113.198:8300 [Candidate] entering Candidate state in term 3
2021/05/07 11:22:26 [INFO] raft: Election won. Tally: 2
2021/05/07 11:22:26 [INFO] raft: Node at 10.242.113.198:8300 [Leader] entering Leader state
2021/05/07 11:22:26 [INFO] raft: Added peer 030af47e-b80c-8935-3357-782656bd05e5, starting replication
2021/05/07 11:22:26 [INFO] consul: cluster leadership acquired
2021/05/07 11:22:26 [INFO] raft: Added peer e90a9b84-7b02-fc1f-15b4-2419fe66fc38, starting replication
2021/05/07 11:22:26 [INFO] consul: New leader elected: dev-consul-server-0
2021/05/07 11:22:26 [INFO] raft: pipelining replication to peer {Voter 030af47e-b80c-8935-3357-782656bd05e5 10.242.139.204:8300}
2021/05/07 11:22:26 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:22:26 [WARN] raft: AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300} rejected, sending older logs (next: 737)
2021/05/07 11:22:26 [INFO] raft: pipelining replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:22:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:48480->10.242.41.137:8301: i/o timeout
2021/05/07 11:22:39 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:22:45 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:23:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:49220->10.242.41.137:8301: i/o timeout
2021/05/07 11:23:39 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:23:41 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:23:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:23:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:45062->10.242.41.146:8301: i/o timeout
2021/05/07 11:23:43 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:23:43 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.502153899s
2021/05/07 11:23:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:49310->10.242.41.137:8301: i/o timeout
2021/05/07 11:23:46 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:23:47 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:23:47 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/07 11:23:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:23:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:49330->10.242.41.137:8301: i/o timeout
2021/05/07 11:23:49 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:23:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:23:50 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:23:50 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:23:50 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:23:52 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500207245s
2021/05/07 11:23:53 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:23:53 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:23:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:23:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:45228->10.242.41.146:8301: i/o timeout
2021/05/07 11:23:55 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:23:56 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500370238s
2021/05/07 11:23:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:23:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:45244->10.242.41.146:8301: i/o timeout
2021/05/07 11:23:57 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:23:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:23:58 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.99584239s
2021/05/07 11:23:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:45288->10.242.41.146:8301: i/o timeout
2021/05/07 11:23:59 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:24:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:01 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.421862115s
2021/05/07 11:24:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:45314->10.242.41.146:8301: i/o timeout
2021/05/07 11:24:02 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:24:04 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:24:04 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:24:04 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:24:04 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:24:04 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:24:04 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:24:05 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:24:05 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/07 11:24:05 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 11:24:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:06 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500223192s
2021/05/07 11:24:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:09 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.997471682s
2021/05/07 11:24:10 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:24:10 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.146:37856
2021/05/07 11:24:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:11 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.458381415s
2021/05/07 11:24:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:14 [INFO] raft: aborting pipeline replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:24:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:14 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:24:14 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:40400->10.242.41.146:8300: i/o timeout
2021/05/07 11:24:15 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:24:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:52450->10.242.41.146:8302: i/o timeout
2021/05/07 11:24:15 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:24:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:16 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500229138s
2021/05/07 11:24:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:19 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.993812475s
2021/05/07 11:24:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:21 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.460768619s
2021/05/07 11:24:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:24 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:41732->10.242.41.146:8300: i/o timeout
2021/05/07 11:24:24 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:24:25 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:41742->10.242.41.146:8300: i/o timeout
2021/05/07 11:24:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:52574->10.242.41.146:8302: i/o timeout
2021/05/07 11:24:25 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:24:26 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:24:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:29 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:24:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:34 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:41858->10.242.41.146:8300: i/o timeout
2021/05/07 11:24:34 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:24:36 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:41884->10.242.41.146:8300: i/o timeout
2021/05/07 11:24:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:36 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:24:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:52762->10.242.41.146:8302: i/o timeout
2021/05/07 11:24:40 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:24:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:43 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:24:43 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:24:44 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:41974->10.242.41.146:8300: i/o timeout
2021/05/07 11:24:44 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:24:44 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:24:44 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:24:44 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:24:45 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:24:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:46 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:42018->10.242.41.146:8300: i/o timeout
2021/05/07 11:24:47 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:24:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:52814->10.242.41.146:8302: i/o timeout
2021/05/07 11:24:50 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:24:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:54 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:42100->10.242.41.146:8300: i/o timeout
2021/05/07 11:24:54 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:24:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:57 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:42156->10.242.41.146:8300: i/o timeout
2021/05/07 11:24:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:24:58 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:25:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:04 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:42222->10.242.41.146:8300: i/o timeout
2021/05/07 11:25:04 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:25:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:08 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:42284->10.242.41.146:8300: i/o timeout
2021/05/07 11:25:09 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:25:09 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:25:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:12 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:25:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:14 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:42342->10.242.41.146:8300: i/o timeout
2021/05/07 11:25:14 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500306934s
2021/05/07 11:25:14 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:25:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:17 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.959265587s
2021/05/07 11:25:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:19 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.418221959s
2021/05/07 11:25:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:22 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:42418->10.242.41.146:8300: i/o timeout
2021/05/07 11:25:23 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:25:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:24 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:25:24 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:25:24 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:42462->10.242.41.146:8300: i/o timeout
2021/05/07 11:25:25 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:25:27 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:25:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:33 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:42572->10.242.41.146:8300: i/o timeout
2021/05/07 11:25:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:34 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:25:35 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:42586->10.242.41.146:8300: i/o timeout
2021/05/07 11:25:35 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:25:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:44 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:42700->10.242.41.146:8300: i/o timeout
2021/05/07 11:25:45 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:25:45 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:42714->10.242.41.146:8300: i/o timeout
2021/05/07 11:25:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:46 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:25:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:49 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:25:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:54 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:25:55 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:42832->10.242.41.146:8300: i/o timeout
2021/05/07 11:25:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:56 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:25:56 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:42862->10.242.41.146:8300: i/o timeout
2021/05/07 11:25:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:25:58 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:25:58 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:25:58 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:25:58 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:25:59 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:25:59 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/07 11:26:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46748->10.242.41.146:8301: i/o timeout
2021/05/07 11:26:01 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:26:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46762->10.242.41.146:8301: i/o timeout
2021/05/07 11:26:04 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:26:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:06 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:42974->10.242.41.146:8300: i/o timeout
2021/05/07 11:26:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:50998->10.242.41.137:8301: i/o timeout
2021/05/07 11:26:07 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:26:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:08 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.113.198:8300
2021/05/07 11:26:09 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:26:09 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:26:09 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:26:09 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:26:09 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:43012->10.242.41.146:8300: i/o timeout
2021/05/07 11:26:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:51044->10.242.41.137:8301: i/o timeout
2021/05/07 11:26:09 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:26:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:10 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500220356s
2021/05/07 11:26:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:12 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:26:12 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:26:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:51076->10.242.41.137:8301: i/o timeout
2021/05/07 11:26:12 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:26:13 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.9336749s
2021/05/07 11:26:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:14 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:26:15 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.348161793s
2021/05/07 11:26:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:18 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:43114->10.242.41.146:8300: i/o timeout
2021/05/07 11:26:19 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:26:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:24 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:43188->10.242.41.146:8300: i/o timeout
2021/05/07 11:26:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:28 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:26:29 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:43260->10.242.41.146:8300: i/o timeout
2021/05/07 11:26:30 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:26:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:34 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:26:34 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:26:34 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:26:34 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:26:34 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:26:34 [WARN] memberlist: Refuting a dead message (from: dev-consul-server-1)
2021/05/07 11:26:36 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:26:37 [ERR] raft: peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300} has newer term, stopping replication
2021/05/07 11:26:37 [INFO] raft: Node at 10.242.113.198:8300 [Follower] entering Follower state (Leader: "")
2021/05/07 11:26:37 [INFO] raft: aborting pipeline replication to peer {Voter 030af47e-b80c-8935-3357-782656bd05e5 10.242.139.204:8300}
2021/05/07 11:26:37 [INFO] consul: cluster leadership lost
2021/05/07 11:26:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47194->10.242.41.146:8301: i/o timeout
2021/05/07 11:26:37 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:26:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47230->10.242.41.146:8301: i/o timeout
2021/05/07 11:26:40 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:26:42 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/05/07 11:26:42 [INFO] raft: Node at 10.242.113.198:8300 [Candidate] entering Candidate state in term 4
2021/05/07 11:26:42 [INFO] raft: Election won. Tally: 2
2021/05/07 11:26:42 [INFO] raft: Node at 10.242.113.198:8300 [Leader] entering Leader state
2021/05/07 11:26:42 [INFO] raft: Added peer 030af47e-b80c-8935-3357-782656bd05e5, starting replication
2021/05/07 11:26:42 [INFO] raft: Added peer e90a9b84-7b02-fc1f-15b4-2419fe66fc38, starting replication
2021/05/07 11:26:42 [INFO] consul: cluster leadership acquired
2021/05/07 11:26:42 [INFO] consul: New leader elected: dev-consul-server-0
2021/05/07 11:26:42 [INFO] raft: pipelining replication to peer {Voter 030af47e-b80c-8935-3357-782656bd05e5 10.242.139.204:8300}
2021/05/07 11:26:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47274->10.242.41.146:8301: i/o timeout
2021/05/07 11:26:43 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:26:44 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:26:44 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:26:45 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.502529832s
2021/05/07 11:26:45 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:26:45 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:26:45 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:26:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:47 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.979991227s
2021/05/07 11:26:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:49 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.113.198:8300
2021/05/07 11:26:50 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.45384957s
2021/05/07 11:26:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:52 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:43424->10.242.41.146:8300: i/o timeout
2021/05/07 11:26:52 [ERR] raft: Failed to make RequestVote RPC to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:43386->10.242.41.146:8300: i/o timeout
2021/05/07 11:26:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:53 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:43548->10.242.41.146:8300: i/o timeout
2021/05/07 11:26:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47412->10.242.41.146:8301: i/o timeout
2021/05/07 11:26:54 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:26:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47428->10.242.41.146:8301: i/o timeout
2021/05/07 11:26:56 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:26:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:26:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47490->10.242.41.146:8301: i/o timeout
2021/05/07 11:26:59 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:27:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:54480->10.242.41.146:8302: i/o timeout
2021/05/07 11:27:00 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:27:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:02 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:43680->10.242.41.146:8300: i/o timeout
2021/05/07 11:27:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47516->10.242.41.146:8301: i/o timeout
2021/05/07 11:27:02 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:27:04 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:43686->10.242.41.146:8300: i/o timeout
2021/05/07 11:27:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:05 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:27:05 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:27:05 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:27:05 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:27:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47554->10.242.41.146:8301: i/o timeout
2021/05/07 11:27:05 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:27:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:12 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:43804->10.242.41.146:8300: i/o timeout
2021/05/07 11:27:12 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:27:14 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:27:14 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:27:14 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:43820->10.242.41.146:8300: i/o timeout
2021/05/07 11:27:15 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:27:15 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:27:15 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/07 11:27:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:54656->10.242.41.146:8302: i/o timeout
2021/05/07 11:27:15 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:27:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:22 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:43930->10.242.41.146:8300: i/o timeout
2021/05/07 11:27:22 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:27:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:25 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8301->10.242.41.146:46702: i/o timeout from=10.242.41.146:46702
2021/05/07 11:27:25 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8301->10.242.41.146:46702: i/o timeout from=10.242.41.146:46702
2021/05/07 11:27:25 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:43958->10.242.41.146:8300: i/o timeout
2021/05/07 11:27:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:54732->10.242.41.146:8302: i/o timeout
2021/05/07 11:27:25 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:27:26 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:27:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:28 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:27:28 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:27:28 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:27:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47868->10.242.41.146:8301: i/o timeout
2021/05/07 11:27:29 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:27:30 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:27:30 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:27:30 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500182833s
2021/05/07 11:27:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47886->10.242.41.146:8301: i/o timeout
2021/05/07 11:27:31 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:27:31 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:27:31 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.146:43038
2021/05/07 11:27:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:32 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:44058->10.242.41.146:8300: i/o timeout
2021/05/07 11:27:32 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.956592608s
2021/05/07 11:27:33 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:27:33 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:27:33 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:27:33 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:27:34 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:27:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47916->10.242.41.146:8301: i/o timeout
2021/05/07 11:27:34 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:27:35 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.436944009s
2021/05/07 11:27:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:38 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:44092->10.242.41.146:8300: i/o timeout
2021/05/07 11:27:39 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:27:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:42 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:44182->10.242.41.146:8300: i/o timeout
2021/05/07 11:27:43 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:27:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:45 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8302->10.242.41.146:49550: i/o timeout from=10.242.41.146:49550
2021/05/07 11:27:45 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8302->10.242.41.146:49550: i/o timeout from=10.242.41.146:49550
2021/05/07 11:27:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:49 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:44268->10.242.41.146:8300: i/o timeout
2021/05/07 11:27:50 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:27:50 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:27:50 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:27:50 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 11:27:50 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/07 11:27:50 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:27:50 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:27:50 [WARN] raft: AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300} rejected, sending older logs (next: 807)
2021/05/07 11:27:50 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:27:50 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:27:50 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:27:51 [INFO] raft: pipelining replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:27:53 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:27:53 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.501812224s
2021/05/07 11:27:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:48160->10.242.41.146:8301: i/o timeout
2021/05/07 11:27:53 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:27:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:27:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:48184->10.242.41.146:8301: i/o timeout
2021/05/07 11:28:01 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:28:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:48268->10.242.41.146:8301: i/o timeout
2021/05/07 11:28:01 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:28:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:52528->10.242.41.137:8301: i/o timeout
2021/05/07 11:28:07 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:28:09 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:28:19 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:28:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:52688->10.242.41.137:8301: i/o timeout
2021/05/07 11:28:19 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:28:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:28:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:28:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:48516->10.242.41.146:8301: i/o timeout
2021/05/07 11:28:22 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:28:23 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500249476s
2021/05/07 11:28:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:52762->10.242.41.137:8301: i/o timeout
2021/05/07 11:28:26 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:28:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:52790->10.242.41.137:8301: i/o timeout
2021/05/07 11:28:29 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:28:30 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:28:30 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:28:30 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:28:31 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:28:32 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500207358s
2021/05/07 11:28:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:28:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:48638->10.242.41.146:8301: i/o timeout
2021/05/07 11:28:32 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:28:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:28:34 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 5.001411606s
2021/05/07 11:28:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:48682->10.242.41.146:8301: i/o timeout
2021/05/07 11:28:35 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:28:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:28:37 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.494812455s
2021/05/07 11:28:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:48702->10.242.41.146:8301: i/o timeout
2021/05/07 11:28:37 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:28:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:28:39 [INFO] raft: aborting pipeline replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:28:39 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:44406->10.242.41.146:8300: i/o timeout
2021/05/07 11:28:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:28:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:55734->10.242.41.146:8302: i/o timeout
2021/05/07 11:28:40 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:28:40 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:28:40 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:28:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:48756->10.242.41.146:8301: i/o timeout
2021/05/07 11:28:40 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:28:41 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:28:41 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:28:42 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500285743s
2021/05/07 11:28:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:28:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:48780->10.242.41.146:8301: i/o timeout
2021/05/07 11:28:42 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:28:43 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:28:43 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:28:43 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:28:43 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:28:44 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:28:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:28:44 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.98029301s
2021/05/07 11:28:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:28:47 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.44169878s
2021/05/07 11:28:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:28:49 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:45028->10.242.41.146:8300: i/o timeout
2021/05/07 11:28:49 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:28:50 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:45040->10.242.41.146:8300: i/o timeout
2021/05/07 11:28:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:28:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:55868->10.242.41.146:8302: i/o timeout
2021/05/07 11:28:50 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:28:51 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:28:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:28:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:28:54 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:28:54 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:28:57 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:28:57 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/07 11:28:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:28:59 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:45164->10.242.41.146:8300: i/o timeout
2021/05/07 11:29:00 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:29:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:00 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:29:00 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.146:45136
2021/05/07 11:29:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:55920->10.242.41.146:8302: i/o timeout
2021/05/07 11:29:00 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:29:01 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:45174->10.242.41.146:8300: i/o timeout
2021/05/07 11:29:01 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:29:01 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:42334
2021/05/07 11:29:02 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:29:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:09 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 11:29:09 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:29:10 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:45282->10.242.41.146:8300: i/o timeout
2021/05/07 11:29:10 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:29:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:10 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 11:29:10 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:29:10 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:29:10 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:29:11 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500217524s
2021/05/07 11:29:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:13 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.955652051s
2021/05/07 11:29:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:16 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.434360172s
2021/05/07 11:29:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:17 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:29:17 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:29:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:19 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:45302->10.242.41.146:8300: i/o timeout
2021/05/07 11:29:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:53422->10.242.41.137:8301: i/o timeout
2021/05/07 11:29:19 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:29:19 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:29:19 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:29:19 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:29:19 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:29:20 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:29:20 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:29:20 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:45400->10.242.41.146:8300: i/o timeout
2021/05/07 11:29:20 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.113.198:8300
2021/05/07 11:29:20 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:29:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:21 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-x5pmr but other probes failed, network may be misconfigured
2021/05/07 11:29:23 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:29:23 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/07 11:29:24 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:29:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:49294->10.242.41.146:8301: i/o timeout
2021/05/07 11:29:24 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:29:25 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 11:29:26 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:29:26 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:29:26 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:29:27 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:29:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:53536->10.242.41.137:8301: i/o timeout
2021/05/07 11:29:27 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:29:28 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500147643s
2021/05/07 11:29:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:30 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:45524->10.242.41.146:8300: i/o timeout
2021/05/07 11:29:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:49374->10.242.41.146:8301: i/o timeout
2021/05/07 11:29:30 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:29:30 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.972615853s
2021/05/07 11:29:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:53588->10.242.41.137:8301: i/o timeout
2021/05/07 11:29:32 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:29:33 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.461376468s
2021/05/07 11:29:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:53600->10.242.41.137:8301: i/o timeout
2021/05/07 11:29:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:29:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:56416->10.242.41.146:8302: i/o timeout
2021/05/07 11:29:35 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:29:36 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:45522->10.242.41.146:8300: i/o timeout
2021/05/07 11:29:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:36 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:29:36 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:29:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:49436->10.242.41.146:8301: i/o timeout
2021/05/07 11:29:36 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:29:37 [ERR] memberlist: Push/Pull with dev-consul-7467bf7b46-x5pmr failed: read tcp 10.242.113.198:53532->10.242.41.137:8301: i/o timeout
2021/05/07 11:29:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:53666->10.242.41.137:8301: i/o timeout
2021/05/07 11:29:38 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:29:39 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:29:39 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:29:39 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:29:39 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:29:40 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:45660->10.242.41.146:8300: i/o timeout
2021/05/07 11:29:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:49498->10.242.41.146:8301: i/o timeout
2021/05/07 11:29:40 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:29:42 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:29:42 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:29:42 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:29:42 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:29:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:45 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 11:29:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:47 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:45752->10.242.41.146:8300: i/o timeout
2021/05/07 11:29:48 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:29:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:50 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:45786->10.242.41.146:8300: i/o timeout
2021/05/07 11:29:50 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:29:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:54 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:29:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:56670->10.242.41.146:8302: i/o timeout
2021/05/07 11:29:55 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:29:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:58 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:45888->10.242.41.146:8300: i/o timeout
2021/05/07 11:29:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:29:59 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:30:00 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:45920->10.242.41.146:8300: i/o timeout
2021/05/07 11:30:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:01 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:30:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:09 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:46016->10.242.41.146:8300: i/o timeout
2021/05/07 11:30:09 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:30:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:56856->10.242.41.146:8302: i/o timeout
2021/05/07 11:30:10 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:30:11 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:46042->10.242.41.146:8300: i/o timeout
2021/05/07 11:30:12 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8301->10.242.41.146:49218: i/o timeout from=10.242.41.146:49218
2021/05/07 11:30:12 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8301->10.242.41.146:49218: i/o timeout from=10.242.41.146:49218
2021/05/07 11:30:12 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:30:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:18 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:30:18 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:30:19 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:46150->10.242.41.146:8300: i/o timeout
2021/05/07 11:30:20 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:30:21 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:30:22 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:30:22 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.146:47044
2021/05/07 11:30:22 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:46176->10.242.41.146:8300: i/o timeout
2021/05/07 11:30:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:25 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:30:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:57028->10.242.41.146:8302: i/o timeout
2021/05/07 11:30:25 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:30:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:30 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:46274->10.242.41.146:8300: i/o timeout
2021/05/07 11:30:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:31 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:30:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:34 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:30:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:35 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:46320->10.242.41.146:8300: i/o timeout
2021/05/07 11:30:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:37 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:30:37 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:43828
2021/05/07 11:30:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:40 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:30:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:57148->10.242.41.146:8302: i/o timeout
2021/05/07 11:30:40 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:30:40 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:30:40 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:30:40 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:30:41 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:46404->10.242.41.146:8300: i/o timeout
2021/05/07 11:30:42 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:30:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:50 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:46540->10.242.41.146:8300: i/o timeout
2021/05/07 11:30:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:51 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-6xm62 10.242.113.229
2021/05/07 11:30:51 [INFO] consul: member 'dev-consul-7467bf7b46-6xm62' joined, marking health alive
2021/05/07 11:30:52 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:46584->10.242.41.146:8300: i/o timeout
2021/05/07 11:30:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:53 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:30:54 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:30:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:30:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:00 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:31:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:00 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:31:00 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:31:03 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:46744->10.242.41.146:8300: i/o timeout
2021/05/07 11:31:03 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:31:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:05 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:31:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:09 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:31:09 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:31:09 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:31:10 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:46890->10.242.41.146:8300: i/o timeout
2021/05/07 11:31:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:50780->10.242.41.146:8301: i/o timeout
2021/05/07 11:31:10 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:31:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:50816->10.242.41.146:8301: i/o timeout
2021/05/07 11:31:13 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:31:15 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:31:15 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:31:15 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:31:15 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:31:15 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:46970->10.242.41.146:8300: i/o timeout
2021/05/07 11:31:19 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:31:20 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:31:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:27 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.113.198:8300
2021/05/07 11:31:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:29 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:47208->10.242.41.146:8300: i/o timeout
2021/05/07 11:31:30 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:47232->10.242.41.146:8300: i/o timeout
2021/05/07 11:31:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:35 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:31:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:41 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:31:42 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:31:42 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:31:43 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:31:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:45 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:47424->10.242.41.146:8300: i/o timeout
2021/05/07 11:31:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:51 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:47550->10.242.41.146:8300: i/o timeout
2021/05/07 11:31:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:56 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:31:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:31:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:32:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:32:01 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:32:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:32:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:32:06 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:47828->10.242.41.146:8300: i/o timeout
2021/05/07 11:32:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:32:07 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:32:07 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.146:47434
2021/05/07 11:32:07 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:32:07 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:32:07 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:32:07 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:32:07 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:32:07 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:32:07 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:32:07 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:32:08 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:32:08 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:32:08 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-x5pmr)
2021/05/07 11:32:08 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:32:08 [ERR] raft: peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300} has newer term, stopping replication
2021/05/07 11:32:08 [INFO] raft: Node at 10.242.113.198:8300 [Follower] entering Follower state (Leader: "")
2021/05/07 11:32:08 [INFO] raft: aborting pipeline replication to peer {Voter 030af47e-b80c-8935-3357-782656bd05e5 10.242.139.204:8300}
2021/05/07 11:32:08 [INFO] consul: cluster leadership lost
2021/05/07 11:32:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:51792->10.242.41.146:8301: i/o timeout
2021/05/07 11:32:08 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:32:14 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/05/07 11:32:14 [INFO] raft: Node at 10.242.113.198:8300 [Candidate] entering Candidate state in term 5
2021/05/07 11:32:14 [INFO] raft: Election won. Tally: 2
2021/05/07 11:32:14 [INFO] raft: Node at 10.242.113.198:8300 [Leader] entering Leader state
2021/05/07 11:32:14 [INFO] raft: Added peer 030af47e-b80c-8935-3357-782656bd05e5, starting replication
2021/05/07 11:32:14 [INFO] raft: Added peer e90a9b84-7b02-fc1f-15b4-2419fe66fc38, starting replication
2021/05/07 11:32:14 [INFO] consul: cluster leadership acquired
2021/05/07 11:32:14 [INFO] consul: New leader elected: dev-consul-server-0
2021/05/07 11:32:14 [INFO] raft: pipelining replication to peer {Voter 030af47e-b80c-8935-3357-782656bd05e5 10.242.139.204:8300}
2021/05/07 11:32:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:51926->10.242.41.146:8301: i/o timeout
2021/05/07 11:32:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:56142->10.242.41.137:8301: i/o timeout
2021/05/07 11:32:15 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:32:17 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.560038516s
2021/05/07 11:32:17 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:32:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:32:19 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 5.073754962s
2021/05/07 11:32:19 [WARN] raft: AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300} rejected, sending older logs (next: 931)
2021/05/07 11:32:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:56228->10.242.41.137:8301: i/o timeout
2021/05/07 11:32:20 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:32:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:32:22 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.50025309s
2021/05/07 11:32:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:32:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:52074->10.242.41.146:8301: i/o timeout
2021/05/07 11:32:23 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:32:24 [ERR] raft: Failed to make RequestVote RPC to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.113.198:47924->10.242.41.146:8300: i/o timeout
2021/05/07 11:32:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:32:24 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:32:24 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:32:25 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 5.340675302s
2021/05/07 11:32:25 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.113.198:48224->10.242.41.146:8300: i/o timeout
2021/05/07 11:32:25 [ERR] raft: peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300} has newer term, stopping replication
2021/05/07 11:32:25 [INFO] raft: Node at 10.242.113.198:8300 [Follower] entering Follower state (Leader: "")
2021/05/07 11:32:25 [INFO] raft: aborting pipeline replication to peer {Voter 030af47e-b80c-8935-3357-782656bd05e5 10.242.139.204:8300}
2021/05/07 11:32:25 [INFO] consul: cluster leadership lost
2021/05/07 11:32:26 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since our last term is greater (5, 4)
2021/05/07 11:32:26 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-x5pmr but other probes failed, network may be misconfigured
2021/05/07 11:32:28 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:32:28 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:32:28 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:32:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:59210->10.242.41.146:8302: i/o timeout
2021/05/07 11:32:30 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:32:32 [INFO] consul: New leader elected: dev-consul-server-2
2021/05/07 11:32:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:59370->10.242.41.146:8302: i/o timeout
2021/05/07 11:32:40 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:32:44 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:32:48 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:32:48 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.146:47926
2021/05/07 11:32:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:59546->10.242.41.146:8302: i/o timeout
2021/05/07 11:32:50 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:32:54 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:32:54 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:32:54 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:32:54 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:32:57 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:32:57 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:44696
2021/05/07 11:33:24 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:33:36 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:33:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:57662->10.242.41.137:8301: i/o timeout
2021/05/07 11:33:40 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:33:42 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:33:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:57696->10.242.41.137:8301: i/o timeout
2021/05/07 11:33:43 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:33:47 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:33:47 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:33:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:60746->10.242.41.146:8302: i/o timeout
2021/05/07 11:33:55 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:34:04 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:34:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:60906->10.242.41.146:8302: i/o timeout
2021/05/07 11:34:05 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:34:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:33000->10.242.41.146:8302: i/o timeout
2021/05/07 11:34:20 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:34:21 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:34:24 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:34:24 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:34:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:58524->10.242.41.137:8301: i/o timeout
2021/05/07 11:34:24 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:34:26 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:34:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:58574->10.242.41.137:8301: i/o timeout
2021/05/07 11:34:27 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:34:35 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:34:35 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:34:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:33398->10.242.41.146:8302: i/o timeout
2021/05/07 11:34:40 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:34:44 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:34:49 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:34:49 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:34:49 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 11:34:50 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 11:34:53 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:35:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:59236->10.242.41.137:8301: i/o timeout
2021/05/07 11:35:00 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:35:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:55080->10.242.41.146:8301: i/o timeout
2021/05/07 11:35:04 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:35:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:55112->10.242.41.146:8301: i/o timeout
2021/05/07 11:35:04 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:35:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:55200->10.242.41.146:8301: i/o timeout
2021/05/07 11:35:07 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:35:09 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:35:09 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:35:13 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:35:13 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:35:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:55370->10.242.41.146:8301: i/o timeout
2021/05/07 11:35:15 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:35:19 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:35:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:55458->10.242.41.146:8301: i/o timeout
2021/05/07 11:35:19 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:35:19 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:35:19 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:35:19 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:35:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:34306->10.242.41.146:8302: i/o timeout
2021/05/07 11:35:25 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:35:26 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:35:26 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:46290
2021/05/07 11:35:29 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:35:29 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:35:29 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:35:30 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:35:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:59858->10.242.41.137:8301: i/o timeout
2021/05/07 11:35:31 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:35:35 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:35:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:34484->10.242.41.146:8302: i/o timeout
2021/05/07 11:35:35 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:35:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:55744->10.242.41.146:8301: i/o timeout
2021/05/07 11:35:35 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:35:36 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:35:36 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:35:36 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:35:41 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:35:41 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:35:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:55894->10.242.41.146:8301: i/o timeout
2021/05/07 11:35:43 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:35:46 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:35:46 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:35:46 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:35:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:55962->10.242.41.146:8301: i/o timeout
2021/05/07 11:35:46 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:35:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:34790->10.242.41.146:8302: i/o timeout
2021/05/07 11:35:50 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:35:59 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:36:09 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 11:36:09 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:36:09 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:36:09 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/07 11:36:09 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:36:19 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since our last index is greater (1147, 1132)
2021/05/07 11:36:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:60776->10.242.41.137:8301: i/o timeout
2021/05/07 11:36:22 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:36:24 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.139.203:42124
2021/05/07 11:36:25 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:36:25 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:36:25 [INFO] consul: New leader elected: dev-consul-server-2
2021/05/07 11:36:26 [DEBUG] raft-net: 10.242.113.198:8300 accepted connection from: 10.242.139.204:49724
2021/05/07 11:36:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:60982->10.242.41.137:8301: i/o timeout
2021/05/07 11:36:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:36:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:32802->10.242.41.137:8301: i/o timeout
2021/05/07 11:36:38 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:36:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:33052->10.242.41.137:8301: i/o timeout
2021/05/07 11:36:50 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:36:52 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:36:52 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:36:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:57156->10.242.41.146:8301: i/o timeout
2021/05/07 11:36:54 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:36:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:35928->10.242.41.146:8302: i/o timeout
2021/05/07 11:36:55 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:36:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:57196->10.242.41.146:8301: i/o timeout
2021/05/07 11:36:56 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:36:57 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:36:57 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:36:57 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:37:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36096->10.242.41.146:8302: i/o timeout
2021/05/07 11:37:05 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:37:08 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:37:08 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:48412
2021/05/07 11:37:13 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:37:13 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:37:13 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 11:37:14 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:37:14 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-x5pmr)
2021/05/07 11:37:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:33738->10.242.41.137:8301: i/o timeout
2021/05/07 11:37:27 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:37:30 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:37:32 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:37:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:33864->10.242.41.137:8301: i/o timeout
2021/05/07 11:37:35 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:37:38 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:37:38 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:37:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:33904->10.242.41.137:8301: i/o timeout
2021/05/07 11:37:39 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:37:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:58048->10.242.41.146:8301: i/o timeout
2021/05/07 11:37:43 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:37:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:58108->10.242.41.146:8301: i/o timeout
2021/05/07 11:37:45 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:37:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:58172->10.242.41.146:8301: i/o timeout
2021/05/07 11:37:48 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:37:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36950->10.242.41.146:8302: i/o timeout
2021/05/07 11:37:50 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:37:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:58212->10.242.41.146:8301: i/o timeout
2021/05/07 11:37:51 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:37:52 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:37:52 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:38:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:37190->10.242.41.146:8302: i/o timeout
2021/05/07 11:38:05 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:38:11 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:38:12 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:38:12 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:49444
2021/05/07 11:38:14 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:38:14 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:38:14 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:38:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:37404->10.242.41.146:8302: i/o timeout
2021/05/07 11:38:15 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:38:24 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:38:32 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:38:32 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:38:39 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:38:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:37936->10.242.41.146:8302: i/o timeout
2021/05/07 11:38:45 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:38:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:38120->10.242.41.146:8302: i/o timeout
2021/05/07 11:38:55 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:39:02 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8302->10.242.41.146:57810: i/o timeout from=10.242.41.146:57810
2021/05/07 11:39:02 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8302->10.242.41.146:57810: i/o timeout from=10.242.41.146:57810
2021/05/07 11:39:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:38278->10.242.41.146:8302: i/o timeout
2021/05/07 11:39:05 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:39:10 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:39:10 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:39:19 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:40:02 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:40:19 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 11:40:29 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:40:34 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:40:34 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:40:34 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 11:40:34 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:40:34 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:40:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:32974->10.242.41.146:8301: i/o timeout
2021/05/07 11:40:37 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:40:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:33006->10.242.41.146:8301: i/o timeout
2021/05/07 11:40:40 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:40:41 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:40:41 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:40:41 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:40:42 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:40:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:37256->10.242.41.137:8301: i/o timeout
2021/05/07 11:40:43 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:40:45 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 11:40:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:37274->10.242.41.137:8301: i/o timeout
2021/05/07 11:40:46 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:40:47 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:40:47 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:40:53 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:41:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:40366->10.242.41.146:8302: i/o timeout
2021/05/07 11:41:00 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:41:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:40546->10.242.41.146:8302: i/o timeout
2021/05/07 11:41:10 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:41:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:40722->10.242.41.146:8302: i/o timeout
2021/05/07 11:41:20 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:41:24 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:41:24 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:41:24 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:41:34 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:42:02 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:42:44 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:43:17 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:43:18 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:43:18 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:43:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:40098->10.242.41.137:8301: i/o timeout
2021/05/07 11:43:20 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:43:23 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:43:23 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:43:24 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-x5pmr but other probes failed, network may be misconfigured
2021/05/07 11:43:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:40216->10.242.41.137:8301: i/o timeout
2021/05/07 11:43:26 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:43:30 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:43:30 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:43:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43082->10.242.41.146:8302: i/o timeout
2021/05/07 11:43:30 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:43:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36096->10.242.41.146:8301: i/o timeout
2021/05/07 11:43:30 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:43:32 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:43:32 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:43:41 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:43:41 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:43:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36350->10.242.41.146:8301: i/o timeout
2021/05/07 11:43:44 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:43:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36374->10.242.41.146:8301: i/o timeout
2021/05/07 11:43:47 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:43:48 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:43:48 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:43:48 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:43:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43452->10.242.41.146:8302: i/o timeout
2021/05/07 11:43:50 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:43:54 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:43:55 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:43:57 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8301->10.242.41.146:57334: i/o timeout from=10.242.41.146:57334
2021/05/07 11:43:57 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8301->10.242.41.146:57334: i/o timeout from=10.242.41.146:57334
2021/05/07 11:44:00 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:44:00 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:44:02 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 11:44:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36734->10.242.41.146:8301: i/o timeout
2021/05/07 11:44:04 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:44:08 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:44:08 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:44:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:41000->10.242.41.137:8301: i/o timeout
2021/05/07 11:44:08 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:44:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36872->10.242.41.146:8301: i/o timeout
2021/05/07 11:44:11 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:44:13 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:44:13 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:44:13 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:44:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43930->10.242.41.146:8302: i/o timeout
2021/05/07 11:44:15 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:44:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:44028->10.242.41.146:8302: i/o timeout
2021/05/07 11:44:25 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:44:27 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:44:27 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:51956
2021/05/07 11:44:40 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 11:44:45 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:44:45 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:44:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:44466->10.242.41.146:8302: i/o timeout
2021/05/07 11:44:45 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:44:47 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:44:47 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:44:47 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 11:44:47 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:44:47 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:44:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:37626->10.242.41.146:8301: i/o timeout
2021/05/07 11:44:55 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:44:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:37744->10.242.41.146:8301: i/o timeout
2021/05/07 11:44:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:41964->10.242.41.137:8301: i/o timeout
2021/05/07 11:44:59 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:45:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:44766->10.242.41.146:8302: i/o timeout
2021/05/07 11:45:01 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:45:01 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:45:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:37808->10.242.41.146:8301: i/o timeout
2021/05/07 11:45:03 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:45:05 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:45:05 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:45:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:44958->10.242.41.146:8302: i/o timeout
2021/05/07 11:45:10 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:45:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:45034->10.242.41.146:8302: i/o timeout
2021/05/07 11:45:20 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:45:30 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:45:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:45310->10.242.41.146:8302: i/o timeout
2021/05/07 11:45:35 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:45:40 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:45:40 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:45:40 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:45:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:45438->10.242.41.146:8302: i/o timeout
2021/05/07 11:45:45 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:46:02 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:46:10 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:46:50 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:47:10 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:47:10 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:47:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:40058->10.242.41.146:8301: i/o timeout
2021/05/07 11:47:14 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:47:16 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:47:16 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 11:47:16 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:47:16 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:47:16 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:47:16 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:47:17 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:47:17 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:47:17 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:47:21 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:47:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:44560->10.242.41.137:8301: i/o timeout
2021/05/07 11:47:30 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:47:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:44634->10.242.41.137:8301: i/o timeout
2021/05/07 11:47:36 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:47:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:40478->10.242.41.146:8301: i/o timeout
2021/05/07 11:47:40 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:47:43 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:47:43 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:47:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:44776->10.242.41.137:8301: i/o timeout
2021/05/07 11:47:43 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:47:44 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:47:44 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:47:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47606->10.242.41.146:8302: i/o timeout
2021/05/07 11:47:45 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:47:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:40620->10.242.41.146:8301: i/o timeout
2021/05/07 11:47:45 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:47:50 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:47:50 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:47:50 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:47:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:44958->10.242.41.137:8301: i/o timeout
2021/05/07 11:47:53 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:47:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:40838->10.242.41.146:8301: i/o timeout
2021/05/07 11:47:56 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:47:56 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:47:56 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:47:56 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:47:58 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:47:58 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:47:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:45072->10.242.41.137:8301: i/o timeout
2021/05/07 11:47:58 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:48:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47896->10.242.41.146:8302: i/o timeout
2021/05/07 11:48:00 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:48:01 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:48:01 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:54224
2021/05/07 11:48:04 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:48:04 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:48:06 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:48:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:45224->10.242.41.137:8301: i/o timeout
2021/05/07 11:48:07 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:48:10 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-x5pmr)
2021/05/07 11:48:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:41208->10.242.41.146:8301: i/o timeout
2021/05/07 11:48:16 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:48:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:41278->10.242.41.146:8301: i/o timeout
2021/05/07 11:48:20 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:48:21 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:48:21 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:48:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:45484->10.242.41.137:8301: i/o timeout
2021/05/07 11:48:22 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:48:22 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:48:22 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:48:24 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:48:24 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:48:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:45516->10.242.41.137:8301: i/o timeout
2021/05/07 11:48:24 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:48:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:48370->10.242.41.146:8302: i/o timeout
2021/05/07 11:48:25 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:48:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:41404->10.242.41.146:8301: i/o timeout
2021/05/07 11:48:26 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:48:30 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:48:30 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:48:30 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:48:30 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:48:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:41502->10.242.41.146:8301: i/o timeout
2021/05/07 11:48:31 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:48:33 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 11:48:35 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 11:48:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:48808->10.242.41.146:8302: i/o timeout
2021/05/07 11:48:50 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:49:00 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:49:00 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 11:49:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46222->10.242.41.137:8301: i/o timeout
2021/05/07 11:49:01 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:49:04 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:49:04 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:49:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46254->10.242.41.137:8301: i/o timeout
2021/05/07 11:49:04 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:49:05 [INFO] consul: New leader elected: dev-consul-server-2
2021/05/07 11:49:05 [DEBUG] raft-net: 10.242.113.198:8300 accepted connection from: 10.242.139.204:35082
2021/05/07 11:49:06 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:49:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46300->10.242.41.137:8301: i/o timeout
2021/05/07 11:49:07 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:49:10 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:49:10 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:49:10 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:49:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:42234->10.242.41.146:8301: i/o timeout
2021/05/07 11:49:10 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:49:15 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 11:49:20 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:49:20 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:49:20 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:49:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:49356->10.242.41.146:8302: i/o timeout
2021/05/07 11:49:20 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:49:32 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:49:40 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:50:02 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 11:50:16 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8301->10.242.41.146:32962: i/o timeout from=10.242.41.146:32962
2021/05/07 11:50:16 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8301->10.242.41.146:32962: i/o timeout from=10.242.41.146:32962
2021/05/07 11:50:32 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:50:33 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8302->10.242.41.146:35798: i/o timeout from=10.242.41.146:35798
2021/05/07 11:50:33 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8302->10.242.41.146:35798: i/o timeout from=10.242.41.146:35798
2021/05/07 11:50:53 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:50:53 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:56288
2021/05/07 11:51:20 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:51:32 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:51:39 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:51:39 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:51:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:52070->10.242.41.146:8302: i/o timeout
2021/05/07 11:51:50 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:52:00 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:52:05 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/07 11:52:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:52326->10.242.41.146:8302: i/o timeout
2021/05/07 11:52:05 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:52:14 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:52:14 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:52:14 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:52:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:52438->10.242.41.146:8302: i/o timeout
2021/05/07 11:52:15 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:52:20 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:52:20 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.146:59738
2021/05/07 11:52:32 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:52:40 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:53:07 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:53:19 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:53:20 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:53:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:50956->10.242.41.137:8301: i/o timeout
2021/05/07 11:53:22 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:53:23 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:53:23 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:53:23 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:53:23 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:53:25 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:53:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:51020->10.242.41.137:8301: i/o timeout
2021/05/07 11:53:25 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:53:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46908->10.242.41.146:8301: i/o timeout
2021/05/07 11:53:28 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:53:30 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:53:30 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:53:30 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:53:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46962->10.242.41.146:8301: i/o timeout
2021/05/07 11:53:31 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:53:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:54182->10.242.41.146:8302: i/o timeout
2021/05/07 11:53:45 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:53:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:54364->10.242.41.146:8302: i/o timeout
2021/05/07 11:53:55 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:53:59 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:53:59 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:57324
2021/05/07 11:54:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:54458->10.242.41.146:8302: i/o timeout
2021/05/07 11:54:05 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:54:10 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:54:10 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:54:10 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:54:21 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.139.204
2021/05/07 11:54:21 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.139.204:8300) (DC: dc1)
2021/05/07 11:54:21 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.139.204
2021/05/07 11:54:21 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.139.204:8300) (DC: dc1)
2021/05/07 11:54:22 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8301->10.242.41.146:34158: i/o timeout from=10.242.41.146:34158
2021/05/07 11:54:22 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8301->10.242.41.146:34158: i/o timeout from=10.242.41.146:34158
2021/05/07 11:54:23 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:54:32 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:54:44 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8302->10.242.41.146:37016: i/o timeout from=10.242.41.146:37016
2021/05/07 11:54:44 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8302->10.242.41.146:37016: i/o timeout from=10.242.41.146:37016
2021/05/07 11:54:47 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:54:47 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:57436
2021/05/07 11:54:48 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8302->10.242.41.146:37032: i/o timeout from=10.242.41.146:37032
2021/05/07 11:54:48 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8302->10.242.41.146:37032: i/o timeout from=10.242.41.146:37032
2021/05/07 11:55:03 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:55:43 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:56:16 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 11:56:21 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 11:56:23 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:56:32 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:56:38 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:56:38 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:56:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:57388->10.242.41.146:8302: i/o timeout
2021/05/07 11:56:45 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:56:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:57568->10.242.41.146:8302: i/o timeout
2021/05/07 11:56:55 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:57:00 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:57:00 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:57778
2021/05/07 11:57:03 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:57:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:57840->10.242.41.146:8302: i/o timeout
2021/05/07 11:57:10 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:57:15 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:57:15 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:57:15 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:57:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:58004->10.242.41.146:8302: i/o timeout
2021/05/07 11:57:20 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:57:32 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:58:05 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:58:13 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:58:53 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:59:33 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:00:02 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:00:13 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:00:35 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8302->10.242.41.146:37664: i/o timeout from=10.242.41.146:37664
2021/05/07 12:00:35 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8302->10.242.41.146:37664: i/o timeout from=10.242.41.146:37664
2021/05/07 12:00:53 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:01:22 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:01:32 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:01:33 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:02:13 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:02:32 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:02:50 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8301->10.242.41.146:35226: i/o timeout from=10.242.41.146:35226
2021/05/07 12:02:50 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8301->10.242.41.146:35226: i/o timeout from=10.242.41.146:35226
2021/05/07 12:02:52 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8301->10.242.41.137:32834: i/o timeout from=10.242.41.137:32834
2021/05/07 12:02:52 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8301->10.242.41.137:32834: i/o timeout from=10.242.41.137:32834
2021/05/07 12:02:53 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:03:33 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:03:33 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:03:35 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:03:35 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:03:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:57842->10.242.41.146:8301: i/o timeout
2021/05/07 12:03:37 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:03:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36644->10.242.41.146:8302: i/o timeout
2021/05/07 12:03:40 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:03:41 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:03:41 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:03:41 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:03:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:57934->10.242.41.146:8301: i/o timeout
2021/05/07 12:03:41 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:03:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36846->10.242.41.146:8302: i/o timeout
2021/05/07 12:03:50 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:04:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:37088->10.242.41.146:8302: i/o timeout
2021/05/07 12:04:05 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:04:10 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 12:04:10 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:04:10 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:04:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:37282->10.242.41.146:8302: i/o timeout
2021/05/07 12:04:15 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:04:32 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:05:12 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:05:12 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:05:13 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/07 12:05:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:38452->10.242.41.146:8302: i/o timeout
2021/05/07 12:05:20 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:05:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:38656->10.242.41.146:8302: i/o timeout
2021/05/07 12:05:30 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:05:33 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:05:40 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 12:05:50 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 12:05:50 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:05:50 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:05:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:39086->10.242.41.146:8302: i/o timeout
2021/05/07 12:05:55 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:06:02 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:06:16 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:06:16 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:58834
2021/05/07 12:06:43 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:07:08 [DEBUG] raft-net: 10.242.113.198:8300 accepted connection from: 10.242.41.146:33704
2021/05/07 12:07:18 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 12:07:21 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:07:21 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:07:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:33650->10.242.41.146:8301: i/o timeout
2021/05/07 12:07:22 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:07:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:33728->10.242.41.146:8301: i/o timeout
2021/05/07 12:07:25 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:07:26 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:07:26 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:07:26 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:07:32 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:08:32 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:09:23 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:09:32 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:09:35 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:09:35 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:09:36 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:09:36 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:09:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36140->10.242.41.146:8301: i/o timeout
2021/05/07 12:09:37 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:09:41 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:09:41 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:09:41 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:09:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36240->10.242.41.146:8301: i/o timeout
2021/05/07 12:09:41 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:09:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43274->10.242.41.146:8302: i/o timeout
2021/05/07 12:09:45 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:09:55 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:09:55 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:59394
2021/05/07 12:10:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43436->10.242.41.146:8302: i/o timeout
2021/05/07 12:10:00 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:10:03 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:10:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43710->10.242.41.146:8302: i/o timeout
2021/05/07 12:10:10 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:10:14 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:10:14 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:10:32 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:10:43 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:11:53 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:12:33 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:13:02 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:13:13 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:14:02 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:14:23 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:15:03 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:17:02 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:17:13 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:18:21 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:18:23 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:19:03 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:19:32 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:19:43 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:20:03 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:20:03 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:20:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47436->10.242.41.146:8301: i/o timeout
2021/05/07 12:20:05 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:20:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47504->10.242.41.146:8301: i/o timeout
2021/05/07 12:20:08 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:20:09 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:20:09 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:20:09 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:20:22 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8301->10.242.41.146:36590: i/o timeout from=10.242.41.146:36590
2021/05/07 12:20:22 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8301->10.242.41.146:36590: i/o timeout from=10.242.41.146:36590
2021/05/07 12:20:32 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:20:32 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:20:32 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:20:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:55142->10.242.41.146:8302: i/o timeout
2021/05/07 12:20:45 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:20:53 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:20:53 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:20:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:48408->10.242.41.146:8301: i/o timeout
2021/05/07 12:20:57 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:20:58 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:20:58 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:20:58 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:21:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:55424->10.242.41.146:8302: i/o timeout
2021/05/07 12:21:00 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:21:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:55482->10.242.41.146:8302: i/o timeout
2021/05/07 12:21:10 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:21:15 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 12:21:15 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:21:15 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:21:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:55770->10.242.41.146:8302: i/o timeout
2021/05/07 12:21:20 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:21:23 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:21:32 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:22:03 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:22:43 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:23:23 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 12:23:23 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:23:25 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:23:25 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:23:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:58128->10.242.41.146:8302: i/o timeout
2021/05/07 12:23:30 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:23:34 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8301->10.242.41.146:36918: i/o timeout from=10.242.41.146:36918
2021/05/07 12:23:34 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8301->10.242.41.146:36918: i/o timeout from=10.242.41.146:36918
2021/05/07 12:23:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:58372->10.242.41.146:8302: i/o timeout
2021/05/07 12:23:45 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:23:49 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:23:49 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:23:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:51508->10.242.41.146:8301: i/o timeout
2021/05/07 12:23:50 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:23:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:51566->10.242.41.146:8301: i/o timeout
2021/05/07 12:23:54 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:23:54 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:23:54 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:23:54 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:23:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:58484->10.242.41.146:8302: i/o timeout
2021/05/07 12:23:55 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:24:00 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 12:24:00 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:24:00 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:24:03 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:24:06 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:24:06 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:24:06 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:24:06 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:24:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:51856->10.242.41.146:8301: i/o timeout
2021/05/07 12:24:09 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:24:12 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:24:12 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:24:12 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:24:14 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-6xm62 10.242.113.229
2021/05/07 12:24:14 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-6xm62 10.242.113.229
2021/05/07 12:24:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:58938->10.242.41.146:8302: i/o timeout
2021/05/07 12:24:15 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:24:18 [ERR] memberlist: Push/Pull with dev-consul-server-1 failed: read tcp 10.242.113.198:51810->10.242.41.146:8301: i/o timeout
2021/05/07 12:24:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:59112->10.242.41.146:8302: i/o timeout
2021/05/07 12:24:25 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:24:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:59308->10.242.41.146:8302: i/o timeout
2021/05/07 12:24:35 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:24:43 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:24:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:59496->10.242.41.146:8302: i/o timeout
2021/05/07 12:24:45 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 12:24:45 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:24:45 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:24:45 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:24:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:56756->10.242.41.137:8301: i/o timeout
2021/05/07 12:24:47 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:24:48 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:24:48 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:24:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:56776->10.242.41.137:8301: i/o timeout
2021/05/07 12:24:50 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:25:43 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:25:54 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:25:54 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:25:54 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:25:54 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:25:54 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:25:54 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 12:25:54 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 12:26:01 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 12:26:03 [INFO] consul: New leader elected: dev-consul-server-2
2021/05/07 12:26:04 [DEBUG] raft-net: 10.242.113.198:8300 accepted connection from: 10.242.139.204:46010
2021/05/07 12:26:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:54036->10.242.41.146:8301: i/o timeout
2021/05/07 12:26:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:54244->10.242.41.146:8301: i/o timeout
2021/05/07 12:26:17 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:26:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:58494->10.242.41.137:8301: i/o timeout
2021/05/07 12:26:21 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:26:21 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:26:21 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:26:21 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:26:23 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:26:23 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:26:23 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:26:23 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-x5pmr but other probes failed, network may be misconfigured
2021/05/07 12:26:24 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:26:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:33104->10.242.41.146:8302: i/o timeout
2021/05/07 12:26:25 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:26:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:54392->10.242.41.146:8301: i/o timeout
2021/05/07 12:26:25 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:26:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:58680->10.242.41.137:8301: i/o timeout
2021/05/07 12:26:29 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:26:29 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:26:29 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:26:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:54508->10.242.41.146:8301: i/o timeout
2021/05/07 12:26:32 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:26:33 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:26:33 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:26:33 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:26:33 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:26:33 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:26:33 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:26:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:54772->10.242.41.146:8301: i/o timeout
2021/05/07 12:26:45 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:26:49 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:26:49 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:26:49 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:26:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:59020->10.242.41.137:8301: i/o timeout
2021/05/07 12:26:49 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:26:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:33612->10.242.41.146:8302: i/o timeout
2021/05/07 12:26:50 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:26:51 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:26:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:59066->10.242.41.137:8301: i/o timeout
2021/05/07 12:26:51 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:26:53 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:27:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:33858->10.242.41.146:8302: i/o timeout
2021/05/07 12:27:05 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:27:09 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:27:09 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:27:10 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:27:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:60004->10.242.41.137:8301: i/o timeout
2021/05/07 12:27:42 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:27:46 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:27:46 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:27:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:60032->10.242.41.137:8301: i/o timeout
2021/05/07 12:27:46 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:27:48 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:27:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:60182->10.242.41.137:8301: i/o timeout
2021/05/07 12:27:52 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:27:54 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:27:54 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:27:54 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:27:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:60302->10.242.41.137:8301: i/o timeout
2021/05/07 12:27:58 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:28:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:60524->10.242.41.137:8301: i/o timeout
2021/05/07 12:28:10 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:28:14 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:28:14 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:28:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:56420->10.242.41.146:8301: i/o timeout
2021/05/07 12:28:15 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:28:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:56442->10.242.41.146:8301: i/o timeout
2021/05/07 12:28:18 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:28:19 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:28:19 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:28:19 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:28:20 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 12:28:33 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:28:33 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:28:33 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:28:33 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/07 12:28:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:32844->10.242.41.137:8301: i/o timeout
2021/05/07 12:28:42 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:28:45 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:28:45 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 12:28:49 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:29:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:33192->10.242.41.137:8301: i/o timeout
2021/05/07 12:29:02 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:29:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:57338->10.242.41.146:8301: i/o timeout
2021/05/07 12:29:07 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:29:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:33318->10.242.41.137:8301: i/o timeout
2021/05/07 12:29:10 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:29:10 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:29:10 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:29:12 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:29:12 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:29:12 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:29:13 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:29:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:57444->10.242.41.146:8301: i/o timeout
2021/05/07 12:29:13 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:29:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36250->10.242.41.146:8302: i/o timeout
2021/05/07 12:29:15 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:29:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:33518->10.242.41.137:8301: i/o timeout
2021/05/07 12:29:18 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:29:19 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:29:19 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:29:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:33538->10.242.41.137:8301: i/o timeout
2021/05/07 12:29:21 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:29:23 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:29:23 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:33548
2021/05/07 12:29:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36542->10.242.41.146:8302: i/o timeout
2021/05/07 12:29:30 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:29:33 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:29:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36794->10.242.41.146:8302: i/o timeout
2021/05/07 12:29:45 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:29:45 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 12:29:45 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:29:45 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:29:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36884->10.242.41.146:8302: i/o timeout
2021/05/07 12:29:55 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:30:06 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:30:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:34470->10.242.41.137:8301: i/o timeout
2021/05/07 12:30:10 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:30:11 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:30:13 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:30:21 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:30:21 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:30:21 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:30:21 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:30:21 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 12:30:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:58730->10.242.41.146:8301: i/o timeout
2021/05/07 12:30:23 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:30:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:58808->10.242.41.146:8301: i/o timeout
2021/05/07 12:30:26 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:30:27 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:30:27 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:30:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:37616->10.242.41.146:8302: i/o timeout
2021/05/07 12:30:35 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:30:36 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/07 12:30:36 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:30:36 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:30:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:59004->10.242.41.146:8301: i/o timeout
2021/05/07 12:30:38 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:30:41 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:30:41 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:30:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:59070->10.242.41.146:8301: i/o timeout
2021/05/07 12:30:41 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:30:47 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:30:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:37976->10.242.41.146:8302: i/o timeout
2021/05/07 12:30:50 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:30:59 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 12:30:59 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:30:59 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:31:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:38048->10.242.41.146:8302: i/o timeout
2021/05/07 12:31:00 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:31:02 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:31:21 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:31:30 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:31:30 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:31:31 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:31:31 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:31:31 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:32:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:60484->10.242.41.146:8301: i/o timeout
2021/05/07 12:32:00 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:32:04 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-x5pmr but other probes failed, network may be misconfigured
2021/05/07 12:32:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:60568->10.242.41.146:8301: i/o timeout
2021/05/07 12:32:07 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:32:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:39416->10.242.41.146:8302: i/o timeout
2021/05/07 12:32:10 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:32:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:60702->10.242.41.146:8301: i/o timeout
2021/05/07 12:32:10 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:32:12 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:32:12 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:32:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36690->10.242.41.137:8301: i/o timeout
2021/05/07 12:32:13 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:32:14 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:32:14 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:32:14 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:32:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:39704->10.242.41.146:8302: i/o timeout
2021/05/07 12:32:25 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:32:31 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:32:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:39826->10.242.41.146:8302: i/o timeout
2021/05/07 12:32:35 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:32:37 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:32:37 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:32:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:32986->10.242.41.146:8301: i/o timeout
2021/05/07 12:32:38 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:32:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:33066->10.242.41.146:8301: i/o timeout
2021/05/07 12:32:42 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:32:43 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:32:43 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:32:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:40278->10.242.41.146:8302: i/o timeout
2021/05/07 12:32:55 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:33:00 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:33:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:37644->10.242.41.137:8301: i/o timeout
2021/05/07 12:33:04 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:33:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:40442->10.242.41.146:8302: i/o timeout
2021/05/07 12:33:05 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:33:06 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8301->10.242.41.146:44860: i/o timeout from=10.242.41.146:44860
2021/05/07 12:33:06 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8301->10.242.41.146:44860: i/o timeout from=10.242.41.146:44860
2021/05/07 12:33:06 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:33:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:37680->10.242.41.137:8301: i/o timeout
2021/05/07 12:33:07 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:33:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:37834->10.242.41.137:8301: i/o timeout
2021/05/07 12:33:12 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:33:14 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:33:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:40560->10.242.41.146:8302: i/o timeout
2021/05/07 12:33:15 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:33:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:37858->10.242.41.137:8301: i/o timeout
2021/05/07 12:33:15 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:33:16 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:33:16 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.146:41460
2021/05/07 12:33:16 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:33:16 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:38624
2021/05/07 12:33:25 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 12:33:25 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:33:25 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:33:29 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:33:29 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:33:29 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:33:29 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:33:34 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-x5pmr)
2021/05/07 12:33:34 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:33:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:38444->10.242.41.137:8301: i/o timeout
2021/05/07 12:33:45 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:33:49 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:33:49 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:33:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:38516->10.242.41.137:8301: i/o timeout
2021/05/07 12:33:49 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:33:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:34498->10.242.41.146:8301: i/o timeout
2021/05/07 12:33:57 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:33:59 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:33:59 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:33:59 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:34:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:34516->10.242.41.146:8301: i/o timeout
2021/05/07 12:34:00 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:34:01 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:34:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:38744->10.242.41.137:8301: i/o timeout
2021/05/07 12:34:03 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:34:04 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:34:04 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:34:06 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:34:06 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:34:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:38826->10.242.41.137:8301: i/o timeout
2021/05/07 12:34:06 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:34:07 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/07 12:34:11 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 12:34:14 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 12:34:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:34884->10.242.41.146:8301: i/o timeout
2021/05/07 12:34:18 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:34:18 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:34:18 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:34:18 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:34:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:41892->10.242.41.146:8302: i/o timeout
2021/05/07 12:34:20 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:34:29 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:34:29 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:34:30 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 12:34:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:35136->10.242.41.146:8301: i/o timeout
2021/05/07 12:34:31 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:34:33 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 12:34:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:35174->10.242.41.146:8301: i/o timeout
2021/05/07 12:34:34 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:34:35 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:34:35 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:34:41 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:34:41 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:40334
2021/05/07 12:34:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:42354->10.242.41.146:8302: i/o timeout
2021/05/07 12:34:45 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:34:46 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:34:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:42468->10.242.41.146:8302: i/o timeout
2021/05/07 12:34:55 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:35:04 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 12:35:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:42808->10.242.41.146:8302: i/o timeout
2021/05/07 12:35:10 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:35:14 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 12:35:14 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:35:14 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:35:18 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8302->10.242.41.146:49162: i/o timeout from=10.242.41.146:49162
2021/05/07 12:35:18 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8302->10.242.41.146:49162: i/o timeout from=10.242.41.146:49162
2021/05/07 12:35:20 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:35:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:42904->10.242.41.146:8302: i/o timeout
2021/05/07 12:35:20 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:35:31 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:35:37 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:35:37 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:35:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36402->10.242.41.146:8301: i/o timeout
2021/05/07 12:35:40 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:35:43 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:35:43 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:35:43 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:35:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:36444->10.242.41.146:8301: i/o timeout
2021/05/07 12:35:43 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:36:00 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:36:31 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:36:40 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:36:41 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:36:41 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:42086
2021/05/07 12:37:05 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 12:37:06 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:37:06 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:37:06 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:37:06 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:37:07 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 12:37:10 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:37:11 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:37:11 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:37:11 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:37:11 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:37:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:42252->10.242.41.137:8301: i/o timeout
2021/05/07 12:37:13 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:37:17 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:37:17 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:37:18 [INFO] consul: New leader elected: dev-consul-server-2
2021/05/07 12:37:18 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 12:37:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:38178->10.242.41.146:8301: i/o timeout
2021/05/07 12:37:20 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:37:21 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:37:21 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:37:21 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:37:22 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:37:22 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:37:22 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:37:25 [DEBUG] raft-net: 10.242.113.198:8300 accepted connection from: 10.242.139.204:58294
2021/05/07 12:37:26 [INFO] consul: New leader elected: dev-consul-server-2
2021/05/07 12:37:28 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 12:37:33 [DEBUG] raft-net: 10.242.113.198:8300 accepted connection from: 10.242.139.204:58426
2021/05/07 12:37:33 [INFO] consul: New leader elected: dev-consul-server-2
2021/05/07 12:37:34 [DEBUG] raft-net: 10.242.113.198:8300 accepted connection from: 10.242.139.204:58452
2021/05/07 12:37:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:42608->10.242.41.137:8301: i/o timeout
2021/05/07 12:37:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:37:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:42680->10.242.41.137:8301: i/o timeout
2021/05/07 12:37:38 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:37:38 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:37:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:38574->10.242.41.146:8301: i/o timeout
2021/05/07 12:37:41 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:37:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:38634->10.242.41.146:8301: i/o timeout
2021/05/07 12:37:44 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:37:45 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:37:45 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:37:45 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:37:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:38678->10.242.41.146:8301: i/o timeout
2021/05/07 12:37:47 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:37:50 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:37:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:45722->10.242.41.146:8302: i/o timeout
2021/05/07 12:37:50 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:38:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:45790->10.242.41.146:8302: i/o timeout
2021/05/07 12:38:00 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:38:03 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:38:03 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.146:45640
2021/05/07 12:38:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46166->10.242.41.146:8302: i/o timeout
2021/05/07 12:38:15 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:38:20 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 12:38:20 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:38:20 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:38:22 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:38:22 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:38:22 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 12:38:23 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:38:23 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:38:23 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/07 12:38:23 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:38:24 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 12:38:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43542->10.242.41.137:8301: i/o timeout
2021/05/07 12:38:24 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:38:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:43594->10.242.41.137:8301: i/o timeout
2021/05/07 12:38:28 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:38:28 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:38:28 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:38:39 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:38:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:39820->10.242.41.146:8301: i/o timeout
2021/05/07 12:38:49 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:38:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:44026->10.242.41.137:8301: i/o timeout
2021/05/07 12:38:53 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:38:54 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:38:54 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:38:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46880->10.242.41.146:8302: i/o timeout
2021/05/07 12:38:55 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:38:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:39980->10.242.41.146:8301: i/o timeout
2021/05/07 12:38:57 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:38:58 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:38:58 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:38:58 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:39:00 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:39:01 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:39:01 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:39:02 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 12:39:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:40050->10.242.41.146:8301: i/o timeout
2021/05/07 12:39:02 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:39:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:40124->10.242.41.146:8301: i/o timeout
2021/05/07 12:39:06 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:39:06 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:39:06 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:39:06 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:39:12 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8301->10.242.41.137:47148: i/o timeout from=10.242.41.137:47148
2021/05/07 12:39:12 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8301->10.242.41.137:47148: i/o timeout from=10.242.41.137:47148
2021/05/07 12:39:17 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/07 12:39:17 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:39:17 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:39:17 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:39:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:40644->10.242.41.146:8301: i/o timeout
2021/05/07 12:39:35 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:39:38 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:39:38 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:39:38 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:39:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:44872->10.242.41.137:8301: i/o timeout
2021/05/07 12:39:39 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:39:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:44970->10.242.41.137:8301: i/o timeout
2021/05/07 12:39:42 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:39:42 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:39:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47802->10.242.41.146:8302: i/o timeout
2021/05/07 12:39:45 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:39:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47976->10.242.41.146:8302: i/o timeout
2021/05/07 12:39:55 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:39:58 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:39:58 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:44850
2021/05/07 12:40:02 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:40:05 [ERR] memberlist: Push/Pull with dev-consul-server-1.dc1 failed: read tcp 10.242.113.198:47648->10.242.41.146:8302: i/o timeout
2021/05/07 12:40:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:48286->10.242.41.146:8302: i/o timeout
2021/05/07 12:40:10 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:40:15 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 12:40:15 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:40:15 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:40:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:48464->10.242.41.146:8302: i/o timeout
2021/05/07 12:40:25 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:40:31 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:40:42 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:41:29 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8301->10.242.41.146:51558: i/o timeout from=10.242.41.146:51558
2021/05/07 12:41:29 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8301->10.242.41.146:51558: i/o timeout from=10.242.41.146:51558
2021/05/07 12:41:31 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:41:33 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:41:52 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:42:19 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:42:19 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.146:49602
2021/05/07 12:42:30 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8302->10.242.41.146:54848: i/o timeout from=10.242.41.146:54848
2021/05/07 12:42:30 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8302->10.242.41.146:54848: i/o timeout from=10.242.41.146:54848
2021/05/07 12:42:30 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8302->10.242.41.146:54850: i/o timeout from=10.242.41.146:54850
2021/05/07 12:42:30 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8302->10.242.41.146:54850: i/o timeout from=10.242.41.146:54850
2021/05/07 12:42:31 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:43:02 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:43:31 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:43:42 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:44:17 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8301->10.242.41.137:50986: i/o timeout from=10.242.41.137:50986
2021/05/07 12:44:17 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8301->10.242.41.137:50986: i/o timeout from=10.242.41.137:50986
2021/05/07 12:44:17 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:44:17 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:44:17 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:44:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:45776->10.242.41.146:8301: i/o timeout
2021/05/07 12:44:18 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:44:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:50006->10.242.41.137:8301: i/o timeout
2021/05/07 12:44:20 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:44:22 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-x5pmr)
2021/05/07 12:44:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:45904->10.242.41.146:8301: i/o timeout
2021/05/07 12:44:24 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:44:25 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:44:25 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:44:25 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 12:44:25 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:44:25 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:44:25 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:44:25 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:44:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:50116->10.242.41.137:8301: i/o timeout
2021/05/07 12:44:27 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:44:39 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:44:39 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:48370
2021/05/07 12:44:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:53146->10.242.41.146:8302: i/o timeout
2021/05/07 12:44:40 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:44:46 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:44:46 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:44:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46348->10.242.41.146:8301: i/o timeout
2021/05/07 12:44:49 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:44:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:53336->10.242.41.146:8302: i/o timeout
2021/05/07 12:44:50 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:44:51 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:44:51 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:44:51 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:44:52 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:44:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:46370->10.242.41.146:8301: i/o timeout
2021/05/07 12:44:52 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:44:58 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:45:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:53538->10.242.41.146:8302: i/o timeout
2021/05/07 12:45:00 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:45:04 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 12:45:04 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:45:04 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:45:31 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:45:32 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:45:45 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.139.204
2021/05/07 12:45:45 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.139.204:8300) (DC: dc1)
2021/05/07 12:45:45 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:45:45 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:45:45 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.139.204
2021/05/07 12:45:45 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.139.204:8300) (DC: dc1)
2021/05/07 12:45:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47432->10.242.41.146:8301: i/o timeout
2021/05/07 12:45:49 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:45:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:47470->10.242.41.146:8301: i/o timeout
2021/05/07 12:45:52 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:45:53 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:45:53 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:45:59 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:45:59 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:46:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:54664->10.242.41.146:8302: i/o timeout
2021/05/07 12:46:05 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:46:09 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8301->10.242.41.146:54496: i/o timeout from=10.242.41.146:54496
2021/05/07 12:46:09 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8301->10.242.41.146:54496: i/o timeout from=10.242.41.146:54496
2021/05/07 12:46:15 [ERR] memberlist: failed to receive: read tcp 10.242.113.198:8302->10.242.41.146:57250: i/o timeout from=10.242.41.146:57250
2021/05/07 12:46:15 [ERR] memberlist: Failed to send error: write tcp 10.242.113.198:8302->10.242.41.146:57250: i/o timeout from=10.242.41.146:57250
2021/05/07 12:46:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:54862->10.242.41.146:8302: i/o timeout
2021/05/07 12:46:15 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:46:19 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:46:20 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-x5pmr but other probes failed, network may be misconfigured
2021/05/07 12:46:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:52242->10.242.41.137:8301: i/o timeout
2021/05/07 12:46:24 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:46:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:54954->10.242.41.146:8302: i/o timeout
2021/05/07 12:46:25 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:46:26 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:46:26 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:46:26 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 12:46:26 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:46:26 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:46:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:52294->10.242.41.137:8301: i/o timeout
2021/05/07 12:46:27 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:46:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:48148->10.242.41.146:8301: i/o timeout
2021/05/07 12:46:30 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:46:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:48228->10.242.41.146:8301: i/o timeout
2021/05/07 12:46:33 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:46:34 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:46:34 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:46:34 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:46:35 [ERR] memberlist: Push/Pull with dev-consul-server-1.dc1 failed: read tcp 10.242.113.198:54700->10.242.41.146:8302: i/o timeout
2021/05/07 12:46:35 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 12:46:35 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:46:35 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:46:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:55230->10.242.41.146:8302: i/o timeout
2021/05/07 12:46:40 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:46:42 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:46:44 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-x5pmr)
2021/05/07 12:46:53 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:46:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:52794->10.242.41.137:8301: i/o timeout
2021/05/07 12:46:54 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:46:55 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:46:55 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:46:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:52870->10.242.41.137:8301: i/o timeout
2021/05/07 12:46:57 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:46:58 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:47:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:52880->10.242.41.137:8301: i/o timeout
2021/05/07 12:47:00 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:47:01 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:47:01 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:47:03 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 12:47:03 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 12:47:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:55762->10.242.41.146:8302: i/o timeout
2021/05/07 12:47:05 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:47:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:48804->10.242.41.146:8301: i/o timeout
2021/05/07 12:47:05 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:47:08 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:47:08 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:47:08 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:47:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:55868->10.242.41.146:8302: i/o timeout
2021/05/07 12:47:15 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:47:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:56252->10.242.41.146:8302: i/o timeout
2021/05/07 12:47:30 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:47:35 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 12:47:35 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:47:35 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:47:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:56302->10.242.41.146:8302: i/o timeout
2021/05/07 12:47:40 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:47:48 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:47:48 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:47:48 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:47:48 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:47:48 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:47:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:49642->10.242.41.146:8301: i/o timeout
2021/05/07 12:47:51 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:48:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:54026->10.242.41.137:8301: i/o timeout
2021/05/07 12:48:02 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:48:04 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 12:48:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:54042->10.242.41.137:8301: i/o timeout
2021/05/07 12:48:06 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 12:48:11 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 12:48:12 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:48:12 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:48:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:57070->10.242.41.146:8302: i/o timeout
2021/05/07 12:48:15 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:48:22 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:48:28 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:48:28 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:51044
2021/05/07 12:48:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:57354->10.242.41.146:8302: i/o timeout
2021/05/07 12:48:30 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:48:34 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 12:48:34 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:48:34 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:48:40 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 12:48:45 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:48:45 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:48:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:57776->10.242.41.146:8302: i/o timeout
2021/05/07 12:48:55 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:49:02 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:49:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.113.198:57920->10.242.41.146:8302: i/o timeout
2021/05/07 12:49:05 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received