By type
+ CONSUL_BIND=
+ [ -n ]
+ CONSUL_CLIENT=
+ [ -n ]
+ CONSUL_DATA_DIR=/consul/data
+ CONSUL_CONFIG_DIR=/consul/config
+ [ -n ]
+ [ a = - ]
+ [ agent = agent ]
+ shift
+ set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
+ [ consul = consul ]
+ stat -c %u /consul/data
+ id -u consul
+ [ 100 != 100 ]
+ stat -c %u /consul/config
+ id -u consul
+ [ 100 != 100 ]
+ [ ! -z ]
+ exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
==> Starting Consul agent...
bootstrap_expect > 0: expecting 3 servers
==> Consul agent running!
Version: 'v1.0.6'
Node ID: '38d8f917-7a55-e6fb-c4e2-f154d8409d0c'
Node name: 'dev-consul-server-1'
Datacenter: 'dc1' (Segment: '')
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.242.146.73 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2021/04/17 10:39:32 [INFO] raft: Initial configuration (index=0): []
2021/04/17 10:39:32 [INFO] raft: Node at 10.242.146.73:8300 [Follower] entering Follower state (Leader: "")
2021/04/17 10:39:32 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.146.73
2021/04/17 10:39:32 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.146.73
2021/04/17 10:39:32 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/17 10:39:32 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2021/04/17 10:39:32 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.146.73:8300) (DC: dc1)
2021/04/17 10:39:32 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2021/04/17 10:39:32 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
2021/04/17 10:39:32 [INFO] agent: started state syncer
2021/04/17 10:39:32 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer
2021/04/17 10:39:32 [INFO] agent: Joining LAN cluster...
2021/04/17 10:39:32 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/04/17 10:39:32 [WARN] memberlist: Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
2021/04/17 10:39:32 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
2021/04/17 10:39:32 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/04/17 10:39:32 [INFO] agent: (LAN) joined: 0 Err: 3 error(s) occurred:
* Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/04/17 10:39:32 [WARN] agent: Join LAN failed: , retrying in 30s
2021/04/17 10:39:39 [WARN] raft: no known peers, aborting election
2021/04/17 10:39:39 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/17 10:39:47 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-zlv6m 10.242.253.199
2021/04/17 10:39:47 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.253.198
2021/04/17 10:39:47 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/04/17 10:39:47 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.253.198
2021/04/17 10:39:47 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.253.198:8300) (DC: dc1)
2021/04/17 10:39:50 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 10:39:50 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 10:39:50 [INFO] consul: Existing Raft peers reported by dev-consul-server-0, disabling bootstrap mode
2021/04/17 10:39:50 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 10:39:50 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=57165018-523c-8d02-02ae-13a7f05d39e4&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2021/04/17 10:39:54 [DEBUG] raft-net: 10.242.146.73:8300 accepted connection from: 10.242.253.198:50672
2021/04/17 10:39:54 [DEBUG] raft-net: 10.242.146.73:8300 accepted connection from: 10.242.253.198:50674
2021/04/17 10:39:54 [WARN] raft: Failed to get previous log: 1 log not found (last: 0)
2021/04/17 10:39:54 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/17 10:39:55 [INFO] agent: Synced node info
2021/04/17 10:40:02 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/04/17 10:40:02 [INFO] agent: (LAN) joined: 3 Err:
2021/04/17 10:40:02 [INFO] agent: Join LAN completed. Synced with 3 initial agents
2021/04/17 11:00:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:60006->10.242.244.80:8301: i/o timeout
2021/04/17 11:00:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:60108->10.242.244.80:8301: i/o timeout
2021/04/17 11:00:15 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:00:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:60156->10.242.244.80:8301: i/o timeout
2021/04/17 11:00:18 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:00:18 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:00:18 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:00:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:46966->10.242.244.80:8302: i/o timeout
2021/04/17 11:00:22 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:00:23 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:00:23 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:00:25 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:00:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:60282->10.242.244.80:8301: i/o timeout
2021/04/17 11:00:28 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:00:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:60320->10.242.244.80:8301: i/o timeout
2021/04/17 11:00:31 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:00:32 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:00:32 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:00:32 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:00:32 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:00:32 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:00:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:60342->10.242.244.80:8301: i/o timeout
2021/04/17 11:00:34 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:00:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:60394->10.242.244.80:8301: i/o timeout
2021/04/17 11:00:37 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:00:38 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:00:38 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:00:38 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:00:41 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:00:41 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:00:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:60474->10.242.244.80:8301: i/o timeout
2021/04/17 11:00:44 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:00:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:60506->10.242.244.80:8301: i/o timeout
2021/04/17 11:00:47 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:00:48 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:00:48 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:00:48 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:00:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:47342->10.242.244.80:8302: i/o timeout
2021/04/17 11:00:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:00:53 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:00:53 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:00:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:60634->10.242.244.80:8301: i/o timeout
2021/04/17 11:00:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:00:59 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:00:59 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:01:02 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:01:02 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:01:02 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:01:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:60770->10.242.244.80:8301: i/o timeout
2021/04/17 11:01:04 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:01:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:60782->10.242.244.80:8301: i/o timeout
2021/04/17 11:01:07 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:01:08 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:01:08 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:01:08 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:01:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:47642->10.242.244.80:8302: i/o timeout
2021/04/17 11:01:12 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:01:17 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 11:01:17 [DEBUG] raft-net: 10.242.146.73:8300 accepted connection from: 10.242.244.80:53894
2021/04/17 11:01:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:47872->10.242.244.80:8302: i/o timeout
2021/04/17 11:01:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:01:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:47932->10.242.244.80:8302: i/o timeout
2021/04/17 11:01:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:01:38 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:01:38 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:01:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:33056->10.242.244.80:8301: i/o timeout
2021/04/17 11:01:40 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:01:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:33074->10.242.244.80:8301: i/o timeout
2021/04/17 11:01:43 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:01:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:33156->10.242.244.80:8301: i/o timeout
2021/04/17 11:01:47 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:01:47 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:01:47 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:01:47 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:01:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:48206->10.242.244.80:8302: i/o timeout
2021/04/17 11:01:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:02:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:48338->10.242.244.80:8302: i/o timeout
2021/04/17 11:02:07 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:02:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:48516->10.242.244.80:8302: i/o timeout
2021/04/17 11:02:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:02:22 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:02:22 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:02:22 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:02:40 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:02:40 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:02:40 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:02:40 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:02:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:33968->10.242.244.80:8301: i/o timeout
2021/04/17 11:02:44 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:02:47 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:02:47 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:02:47 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:02:47 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:02:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:34008->10.242.244.80:8301: i/o timeout
2021/04/17 11:02:47 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:02:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:49082->10.242.244.80:8302: i/o timeout
2021/04/17 11:02:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:03:00 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 11:03:03 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:03:03 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:03:03 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:03:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:34262->10.242.244.80:8301: i/o timeout
2021/04/17 11:03:06 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:03:08 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/17 11:03:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:34398->10.242.244.80:8301: i/o timeout
2021/04/17 11:03:14 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:03:16 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:03:16 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:03:16 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:03:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:49430->10.242.244.80:8302: i/o timeout
2021/04/17 11:03:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:03:20 [WARN] raft: Heartbeat timeout from "10.242.253.198:8300" reached, starting election
2021/04/17 11:03:20 [INFO] raft: Node at 10.242.146.73:8300 [Candidate] entering Candidate state in term 3
2021/04/17 11:03:20 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:03:20 [INFO] raft: Election won. Tally: 2
2021/04/17 11:03:20 [INFO] raft: Node at 10.242.146.73:8300 [Leader] entering Leader state
2021/04/17 11:03:20 [INFO] raft: Added peer 79007099-56a4-82c8-2a6b-8367b805feea, starting replication
2021/04/17 11:03:20 [INFO] raft: Added peer e3d1abd7-c4ef-d7bc-57e1-b271f273a71b, starting replication
2021/04/17 11:03:20 [INFO] consul: cluster leadership acquired
2021/04/17 11:03:20 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:03:20 [INFO] consul: New leader elected: dev-consul-server-1
2021/04/17 11:03:20 [INFO] raft: pipelining replication to peer {Voter e3d1abd7-c4ef-d7bc-57e1-b271f273a71b 10.242.253.198:8300}
2021/04/17 11:03:20 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:03:21 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:03:23 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.50226499s
2021/04/17 11:03:23 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:03:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:03:25 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.961017835s
2021/04/17 11:03:26 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.146.73:8300
2021/04/17 11:03:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:03:27 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:03:27 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/17 11:03:28 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500308035s
2021/04/17 11:03:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:03:28 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.146.73:8300
2021/04/17 11:03:29 [ERR] raft: peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300} has newer term, stopping replication
2021/04/17 11:03:29 [INFO] raft: Node at 10.242.146.73:8300 [Follower] entering Follower state (Leader: "")
2021/04/17 11:03:29 [INFO] raft: aborting pipeline replication to peer {Voter e3d1abd7-c4ef-d7bc-57e1-b271f273a71b 10.242.253.198:8300}
2021/04/17 11:03:29 [INFO] consul: cluster leadership lost
2021/04/17 11:03:29 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:03:29 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:03:34 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/17 11:03:34 [INFO] raft: Node at 10.242.146.73:8300 [Candidate] entering Candidate state in term 4
2021/04/17 11:03:34 [INFO] raft: Node at 10.242.146.73:8300 [Follower] entering Follower state (Leader: "")
2021/04/17 11:03:36 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since our last term is greater (3, 2)
2021/04/17 11:03:40 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/17 11:03:40 [INFO] raft: Node at 10.242.146.73:8300 [Candidate] entering Candidate state in term 8
2021/04/17 11:03:40 [INFO] raft: Election won. Tally: 2
2021/04/17 11:03:40 [INFO] raft: Node at 10.242.146.73:8300 [Leader] entering Leader state
2021/04/17 11:03:40 [INFO] raft: Added peer 79007099-56a4-82c8-2a6b-8367b805feea, starting replication
2021/04/17 11:03:40 [INFO] raft: Added peer e3d1abd7-c4ef-d7bc-57e1-b271f273a71b, starting replication
2021/04/17 11:03:40 [INFO] consul: cluster leadership acquired
2021/04/17 11:03:40 [INFO] consul: New leader elected: dev-consul-server-1
2021/04/17 11:03:40 [INFO] raft: pipelining replication to peer {Voter e3d1abd7-c4ef-d7bc-57e1-b271f273a71b 10.242.253.198:8300}
2021/04/17 11:03:40 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:03:40 [WARN] raft: AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300} rejected, sending older logs (next: 338)
2021/04/17 11:03:40 [INFO] raft: pipelining replication to peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}
2021/04/17 11:04:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:35256->10.242.244.80:8301: i/o timeout
2021/04/17 11:04:20 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:04:21 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:04:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:04:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:35278->10.242.244.80:8301: i/o timeout
2021/04/17 11:04:23 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:04:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:04:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:35414->10.242.244.80:8301: i/o timeout
2021/04/17 11:04:31 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:04:31 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:04:32 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500243717s
2021/04/17 11:04:35 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:04:35 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:04:55 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:04:55 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 11:04:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:04:56 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500353353s
2021/04/17 11:04:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:35750->10.242.244.80:8301: i/o timeout
2021/04/17 11:04:58 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:05:01 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:05:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:05:02 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500230226s
2021/04/17 11:05:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:35816->10.242.244.80:8301: i/o timeout
2021/04/17 11:05:03 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:05:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:05:04 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.954356645s
2021/04/17 11:05:06 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/17 11:05:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:05:07 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/17 11:05:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:05:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:35900->10.242.244.80:8301: i/o timeout
2021/04/17 11:05:09 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500296404s
2021/04/17 11:05:10 [INFO] raft: aborting pipeline replication to peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}
2021/04/17 11:05:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:05:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:05:12 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500289008s
2021/04/17 11:05:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:35954->10.242.244.80:8301: i/o timeout
2021/04/17 11:05:13 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:05:13 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.146.73:8300
2021/04/17 11:05:13 [ERR] raft: peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300} has newer term, stopping replication
2021/04/17 11:05:13 [INFO] raft: Node at 10.242.146.73:8300 [Follower] entering Follower state (Leader: "")
2021/04/17 11:05:13 [INFO] raft: aborting pipeline replication to peer {Voter e3d1abd7-c4ef-d7bc-57e1-b271f273a71b 10.242.253.198:8300}
2021/04/17 11:05:13 [INFO] consul: cluster leadership lost
2021/04/17 11:05:14 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:05:14 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:05:14 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:05:17 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:05:17 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:05:22 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since our last index is greater (402, 398)
2021/04/17 11:05:22 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/17 11:05:22 [INFO] raft: Node at 10.242.146.73:8300 [Candidate] entering Candidate state in term 11
2021/04/17 11:05:22 [INFO] raft: Election won. Tally: 2
2021/04/17 11:05:22 [INFO] raft: Node at 10.242.146.73:8300 [Leader] entering Leader state
2021/04/17 11:05:22 [INFO] raft: Added peer 79007099-56a4-82c8-2a6b-8367b805feea, starting replication
2021/04/17 11:05:22 [INFO] raft: Added peer e3d1abd7-c4ef-d7bc-57e1-b271f273a71b, starting replication
2021/04/17 11:05:22 [INFO] consul: cluster leadership acquired
2021/04/17 11:05:22 [INFO] consul: New leader elected: dev-consul-server-1
2021/04/17 11:05:22 [INFO] raft: pipelining replication to peer {Voter e3d1abd7-c4ef-d7bc-57e1-b271f273a71b 10.242.253.198:8300}
2021/04/17 11:05:23 [WARN] raft: AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300} rejected, sending older logs (next: 399)
2021/04/17 11:05:23 [INFO] raft: pipelining replication to peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}
2021/04/17 11:05:29 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.50016921s
2021/04/17 11:05:29 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:05:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:36172->10.242.244.80:8301: i/o timeout
2021/04/17 11:05:30 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:05:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:05:32 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.979611122s
2021/04/17 11:05:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:51192->10.242.244.80:8302: i/o timeout
2021/04/17 11:05:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:05:32 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:05:32 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:05:32 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:05:32 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:05:32 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:05:32 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:05:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:05:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:05:35 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500312222s
2021/04/17 11:05:43 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:05:43 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:05:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:05:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:36482->10.242.244.80:8301: i/o timeout
2021/04/17 11:05:46 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:05:46 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500229916s
2021/04/17 11:05:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:05:48 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:05:48 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:05:48 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:05:48 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:05:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:05:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:36514->10.242.244.80:8301: i/o timeout
2021/04/17 11:05:49 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:05:49 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500222504s
2021/04/17 11:05:49 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:05:49 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:05:49 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:05:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:05:52 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500272912s
2021/04/17 11:05:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:05:53 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/17 11:05:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:36590->10.242.244.80:8301: i/o timeout
2021/04/17 11:05:54 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:05:55 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:05:55 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:05:55 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:05:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:05:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:36628->10.242.244.80:8301: i/o timeout
2021/04/17 11:05:58 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:05:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:06:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:06:03 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:06:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:36714->10.242.244.80:8301: i/o timeout
2021/04/17 11:06:04 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:06:09 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:06:09 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:06:10 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500269267s
2021/04/17 11:06:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:06:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:36810->10.242.244.80:8301: i/o timeout
2021/04/17 11:06:11 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:06:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:06:13 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.961813709s
2021/04/17 11:06:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:36848->10.242.244.80:8301: i/o timeout
2021/04/17 11:06:14 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:06:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:06:15 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:06:15 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:06:15 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:06:15 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:06:15 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:06:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:06:17 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500323911s
2021/04/17 11:06:18 [INFO] raft: aborting pipeline replication to peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}
2021/04/17 11:06:18 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:06:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:06:20 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:06:20 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:06:20 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:06:20 [INFO] raft: pipelining replication to peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}
2021/04/17 11:06:23 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:06:23 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:06:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:06:29 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:06:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:06:33 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:06:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:37116->10.242.244.80:8301: i/o timeout
2021/04/17 11:06:34 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:06:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:06:37 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:06:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:06:43 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:06:43 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:06:45 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:06:45 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:06:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:37276->10.242.244.80:8301: i/o timeout
2021/04/17 11:06:46 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:06:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:06:48 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500256247s
2021/04/17 11:06:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:06:50 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:06:50 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:06:50 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:06:50 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:06:50 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:06:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:06:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:37346->10.242.244.80:8301: i/o timeout
2021/04/17 11:06:51 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:06:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:52342->10.242.244.80:8302: i/o timeout
2021/04/17 11:06:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:06:52 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500188655s
2021/04/17 11:06:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:06:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:06:55 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.942211003s
2021/04/17 11:06:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:06:57 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 7.40474182s
2021/04/17 11:06:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:00 [INFO] raft: aborting pipeline replication to peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}
2021/04/17 11:07:00 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:07:00 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:07:00 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:55846
2021/04/17 11:07:00 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:51196->10.242.244.80:8300: i/o timeout
2021/04/17 11:07:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:01 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:07:02 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.50029046s
2021/04/17 11:07:03 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:07:03 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 11:07:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:05 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.951168427s
2021/04/17 11:07:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:52534->10.242.244.80:8302: i/o timeout
2021/04/17 11:07:07 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:07:07 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 7.41497007s
2021/04/17 11:07:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:10 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:54130->10.242.244.80:8300: i/o timeout
2021/04/17 11:07:10 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:07:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:11 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:54154->10.242.244.80:8300: i/o timeout
2021/04/17 11:07:12 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:07:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:52608->10.242.244.80:8302: i/o timeout
2021/04/17 11:07:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:07:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:20 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:54266->10.242.244.80:8300: i/o timeout
2021/04/17 11:07:20 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:07:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:22 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:07:22 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:07:22 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:07:22 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:54306->10.242.244.80:8300: i/o timeout
2021/04/17 11:07:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:23 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:07:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:52802->10.242.244.80:8302: i/o timeout
2021/04/17 11:07:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:07:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:30 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:54386->10.242.244.80:8300: i/o timeout
2021/04/17 11:07:30 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:07:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:32 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:07:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:33 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:54438->10.242.244.80:8300: i/o timeout
2021/04/17 11:07:34 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:07:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:40 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:54526->10.242.244.80:8300: i/o timeout
2021/04/17 11:07:40 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:07:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:42 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:07:42 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:07:43 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:07:44 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:54582->10.242.244.80:8300: i/o timeout
2021/04/17 11:07:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:45 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:07:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:50 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:54650->10.242.244.80:8300: i/o timeout
2021/04/17 11:07:50 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:07:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:54 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:07:54 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:07:54 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:07:54 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:07:54 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:07:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:07:56 [INFO] raft: pipelining replication to peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}
2021/04/17 11:08:21 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:08:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:08:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:38588->10.242.244.80:8301: i/o timeout
2021/04/17 11:08:23 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:08:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:08:25 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:08:25 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:08:25 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:08:26 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:08:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:08:27 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500378456s
2021/04/17 11:08:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:08:29 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:08:29 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:08:29 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:08:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:38690->10.242.244.80:8301: i/o timeout
2021/04/17 11:08:31 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:08:31 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:08:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:08:32 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500224512s
2021/04/17 11:08:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:38730->10.242.244.80:8301: i/o timeout
2021/04/17 11:08:34 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:08:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:08:35 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:08:35 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:08:35 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:08:35 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:08:35 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:08:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:08:36 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:08:38 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500322936s
2021/04/17 11:08:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:08:40 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:08:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:08:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:53922->10.242.244.80:8302: i/o timeout
2021/04/17 11:08:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:08:42 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:08:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:08:44 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:08:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:08:45 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:08:45 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:08:45 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:08:45 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:08:49 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:08:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:39068->10.242.244.80:8301: i/o timeout
2021/04/17 11:08:52 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:08:53 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:08:53 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500313322s
2021/04/17 11:08:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:08:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:39096->10.242.244.80:8301: i/o timeout
2021/04/17 11:08:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:08:56 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.973286589s
2021/04/17 11:08:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:08:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:39138->10.242.244.80:8301: i/o timeout
2021/04/17 11:08:58 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:08:58 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 7.470735754s
2021/04/17 11:08:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:09:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:39160->10.242.244.80:8301: i/o timeout
2021/04/17 11:09:01 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:09:02 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:09:02 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:09:02 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:09:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:54180->10.242.244.80:8302: i/o timeout
2021/04/17 11:09:02 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:09:02 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:09:03 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:09:04 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500325699s
2021/04/17 11:09:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:09:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:09:07 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.962698396s
2021/04/17 11:09:08 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:09:09 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:09:09 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:09:11 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:09:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:54308->10.242.244.80:8302: i/o timeout
2021/04/17 11:09:12 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:09:14 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:09:15 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:09:15 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:09:15 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:09:16 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:09:16 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:09:16 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:09:16 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:09:19 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:09:19 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:09:19 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 11:09:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:09:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:39442->10.242.244.80:8301: i/o timeout
2021/04/17 11:09:21 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:09:21 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500319048s
2021/04/17 11:09:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:09:23 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:09:23 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:09:23 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:09:24 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.92395835s
2021/04/17 11:09:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:39458->10.242.244.80:8301: i/o timeout
2021/04/17 11:09:24 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:09:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:09:26 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 7.405050629s
2021/04/17 11:09:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:09:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:54506->10.242.244.80:8302: i/o timeout
2021/04/17 11:09:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:09:27 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:54732->10.242.244.80:8300: i/o timeout
2021/04/17 11:09:28 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:09:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:09:29 [INFO] raft: aborting pipeline replication to peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}
2021/04/17 11:09:29 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:09:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:09:31 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500262406s
2021/04/17 11:09:32 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:09:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:09:34 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.966768856s
2021/04/17 11:09:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:09:36 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 7.437284518s
2021/04/17 11:09:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:09:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:54642->10.242.244.80:8302: i/o timeout
2021/04/17 11:09:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:09:38 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:56208->10.242.244.80:8300: i/o timeout
2021/04/17 11:09:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:09:39 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:09:39 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:56218->10.242.244.80:8300: i/o timeout
2021/04/17 11:09:39 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:09:40 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:09:40 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:59142
2021/04/17 11:09:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:09:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:09:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:09:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:09:47 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:09:47 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:09:47 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:09:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:09:49 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:56356->10.242.244.80:8300: i/o timeout
2021/04/17 11:09:49 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:56358->10.242.244.80:8300: i/o timeout
2021/04/17 11:09:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:39810->10.242.244.80:8301: i/o timeout
2021/04/17 11:09:49 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:09:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:09:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:54840->10.242.244.80:8302: i/o timeout
2021/04/17 11:09:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:09:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:09:53 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:09:53 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:09:53 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:09:53 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:09:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:09:56 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:09:56 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:09:57 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:09:57 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:09:57 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:09:57 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:09:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:09:59 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:56488->10.242.244.80:8300: i/o timeout
2021/04/17 11:09:59 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:09:59 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:56494->10.242.244.80:8300: i/o timeout
2021/04/17 11:10:00 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:10:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:02 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/17 11:10:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:09 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:56606->10.242.244.80:8300: i/o timeout
2021/04/17 11:10:09 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:10:10 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:10:10 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:10:10 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:56628->10.242.244.80:8300: i/o timeout
2021/04/17 11:10:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:11 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:10:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:19 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:56736->10.242.244.80:8300: i/o timeout
2021/04/17 11:10:19 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:10:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:21 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:56768->10.242.244.80:8300: i/o timeout
2021/04/17 11:10:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:55204->10.242.244.80:8302: i/o timeout
2021/04/17 11:10:22 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:10:22 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:10:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:29 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:56856->10.242.244.80:8300: i/o timeout
2021/04/17 11:10:29 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:10:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:55342->10.242.244.80:8302: i/o timeout
2021/04/17 11:10:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:10:32 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:56902->10.242.244.80:8300: i/o timeout
2021/04/17 11:10:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:33 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:10:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:36 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:10:36 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:10:37 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:10:37 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:60878
2021/04/17 11:10:39 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:56996->10.242.244.80:8300: i/o timeout
2021/04/17 11:10:39 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:10:39 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:10:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:41 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:10:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:43 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:57056->10.242.244.80:8300: i/o timeout
2021/04/17 11:10:44 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:10:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:47 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:10:47 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:10:47 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:10:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:55540->10.242.244.80:8302: i/o timeout
2021/04/17 11:10:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:10:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:49 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:10:49 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:10:49 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:10:49 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:57120->10.242.244.80:8300: i/o timeout
2021/04/17 11:10:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:40612->10.242.244.80:8301: i/o timeout
2021/04/17 11:10:51 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:10:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:54 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:10:54 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:57194->10.242.244.80:8300: i/o timeout
2021/04/17 11:10:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:40662->10.242.244.80:8301: i/o timeout
2021/04/17 11:10:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:10:55 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:10:55 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:10:55 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:10:55 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:10:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:10:56 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:10:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:00 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:57280->10.242.244.80:8300: i/o timeout
2021/04/17 11:11:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:01 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:11:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:06 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:57364->10.242.244.80:8300: i/o timeout
2021/04/17 11:11:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:10 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:11:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:11 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:57422->10.242.244.80:8300: i/o timeout
2021/04/17 11:11:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:14 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:11:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:18 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:11:18 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:11:19 [ERR] memberlist: failed to receive: read tcp 10.242.146.73:8302->10.242.244.80:52406: i/o timeout from=10.242.244.80:52406
2021/04/17 11:11:19 [ERR] memberlist: Failed to send error: write tcp 10.242.146.73:8302->10.242.244.80:52406: i/o timeout from=10.242.244.80:52406
2021/04/17 11:11:20 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:57536->10.242.244.80:8300: i/o timeout
2021/04/17 11:11:21 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:11:21 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:11:21 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 11:11:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:24 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:57606->10.242.244.80:8300: i/o timeout
2021/04/17 11:11:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:26 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:11:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:29 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:11:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:36 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:57766->10.242.244.80:8300: i/o timeout
2021/04/17 11:11:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:39 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:57804->10.242.244.80:8300: i/o timeout
2021/04/17 11:11:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:47 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:11:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:49 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:11:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:57 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:58120->10.242.244.80:8300: i/o timeout
2021/04/17 11:11:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:11:59 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:58154->10.242.244.80:8300: i/o timeout
2021/04/17 11:12:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:00 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:12:00 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:12:03 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:12:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:08 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:12:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:09 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:12:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:18 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:58404->10.242.244.80:8300: i/o timeout
2021/04/17 11:12:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:19 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:58426->10.242.244.80:8300: i/o timeout
2021/04/17 11:12:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:21 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:12:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:29 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:12:30 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:12:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:39 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:58666->10.242.244.80:8300: i/o timeout
2021/04/17 11:12:40 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:58680->10.242.244.80:8300: i/o timeout
2021/04/17 11:12:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:42 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:12:42 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:12:45 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:12:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:50 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:12:50 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:12:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:54 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:12:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:12:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:00 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:58952->10.242.244.80:8300: i/o timeout
2021/04/17 11:13:00 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:58954->10.242.244.80:8300: i/o timeout
2021/04/17 11:13:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:09 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:13:09 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:13:09 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:13:09 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.253.198
2021/04/17 11:13:09 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/04/17 11:13:09 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:13:09 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:13:10 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.253.198
2021/04/17 11:13:10 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/04/17 11:13:10 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:13:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:42640->10.242.244.80:8301: i/o timeout
2021/04/17 11:13:13 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:13:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:15 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:13:15 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:13:15 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:13:15 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:13:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:57694->10.242.244.80:8302: i/o timeout
2021/04/17 11:13:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:13:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:20 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:59278->10.242.244.80:8300: i/o timeout
2021/04/17 11:13:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:21 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:59306->10.242.244.80:8300: i/o timeout
2021/04/17 11:13:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:24 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:13:24 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:13:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:57840->10.242.244.80:8302: i/o timeout
2021/04/17 11:13:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:13:27 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:13:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:30 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:13:32 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:13:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:57986->10.242.244.80:8302: i/o timeout
2021/04/17 11:13:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:13:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:40 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:59590->10.242.244.80:8300: i/o timeout
2021/04/17 11:13:42 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:59606->10.242.244.80:8300: i/o timeout
2021/04/17 11:13:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:58122->10.242.244.80:8302: i/o timeout
2021/04/17 11:13:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:13:47 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:13:47 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:13:47 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:13:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:51 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:13:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:53 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:13:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:13:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:01 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:14:01 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:59876->10.242.244.80:8300: i/o timeout
2021/04/17 11:14:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:03 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:59890->10.242.244.80:8300: i/o timeout
2021/04/17 11:14:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:11 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:14:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:14 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:14:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:21 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:60144->10.242.244.80:8300: i/o timeout
2021/04/17 11:14:22 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:14:22 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:34174
2021/04/17 11:14:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:24 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:60178->10.242.244.80:8300: i/o timeout
2021/04/17 11:14:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:31 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:14:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:34 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:14:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:40 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:14:40 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:14:41 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:14:41 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:60438->10.242.244.80:8300: i/o timeout
2021/04/17 11:14:41 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:14:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:44 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:60490->10.242.244.80:8300: i/o timeout
2021/04/17 11:14:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:48 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:14:48 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:14:48 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:14:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:48 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:14:48 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/04/17 11:14:48 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:14:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:44176->10.242.244.80:8301: i/o timeout
2021/04/17 11:14:53 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:14:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:55 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:14:55 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:14:55 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:14:55 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:14:56 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:14:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:14:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:01 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:60844->10.242.244.80:8300: i/o timeout
2021/04/17 11:15:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:59220->10.242.244.80:8302: i/o timeout
2021/04/17 11:15:02 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:15:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:06 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:60900->10.242.244.80:8300: i/o timeout
2021/04/17 11:15:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:12 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:15:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:16 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:15:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:59410->10.242.244.80:8302: i/o timeout
2021/04/17 11:15:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:15:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:19 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:15:19 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:15:19 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:15:21 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:15:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:44564->10.242.244.80:8301: i/o timeout
2021/04/17 11:15:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:15:22 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500385021s
2021/04/17 11:15:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:24 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.981752927s
2021/04/17 11:15:25 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:15:25 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:15:25 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:15:25 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:15:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:44592->10.242.244.80:8301: i/o timeout
2021/04/17 11:15:25 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:15:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:26 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:15:28 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500204589s
2021/04/17 11:15:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:29 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:15:29 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:32876->10.242.244.80:8300: i/o timeout
2021/04/17 11:15:29 [INFO] raft: pipelining replication to peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}
2021/04/17 11:15:31 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:15:31 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:15:31 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 11:15:32 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/17 11:15:32 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.50025612s
2021/04/17 11:15:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:35 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.987663427s
2021/04/17 11:15:37 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 7.418942841s
2021/04/17 11:15:37 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:15:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:39 [INFO] raft: aborting pipeline replication to peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}
2021/04/17 11:15:39 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:15:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:41 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:32936->10.242.244.80:8300: i/o timeout
2021/04/17 11:15:41 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:15:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:59826->10.242.244.80:8302: i/o timeout
2021/04/17 11:15:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:15:42 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500371388s
2021/04/17 11:15:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:44 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.966737568s
2021/04/17 11:15:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:47 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 7.436047027s
2021/04/17 11:15:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:49 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:33248->10.242.244.80:8300: i/o timeout
2021/04/17 11:15:49 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:15:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:51 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:33292->10.242.244.80:8300: i/o timeout
2021/04/17 11:15:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:59966->10.242.244.80:8302: i/o timeout
2021/04/17 11:15:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:15:52 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:15:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:15:59 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:15:59 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:36112
2021/04/17 11:15:59 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:15:59 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:15:59 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:33390->10.242.244.80:8300: i/o timeout
2021/04/17 11:15:59 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:16:01 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:16:01 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:16:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:60042->10.242.244.80:8302: i/o timeout
2021/04/17 11:16:02 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:16:02 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:33430->10.242.244.80:8300: i/o timeout
2021/04/17 11:16:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:16:03 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:16:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:16:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:16:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:16:09 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:33536->10.242.244.80:8300: i/o timeout
2021/04/17 11:16:10 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:16:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:16:12 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:16:12 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:16:12 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:16:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:16:13 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:33582->10.242.244.80:8300: i/o timeout
2021/04/17 11:16:13 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:16:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:16:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:16:17 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:16:17 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:16:17 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:16:17 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:16:17 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:16:17 [INFO] raft: pipelining replication to peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}
2021/04/17 11:16:17 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:18:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:46782->10.242.244.80:8301: i/o timeout
2021/04/17 11:18:02 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:18:03 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:18:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:18:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:46808->10.242.244.80:8301: i/o timeout
2021/04/17 11:18:05 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:18:06 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500286125s
2021/04/17 11:18:06 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:18:06 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:18:06 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:18:06 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:18:06 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:18:06 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:18:06 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:18:07 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:18:09 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:18:10 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500318451s
2021/04/17 11:18:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:18:11 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/17 11:18:12 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:18:13 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:18:17 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:18:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:46994->10.242.244.80:8301: i/o timeout
2021/04/17 11:18:18 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:18:18 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500331556s
2021/04/17 11:18:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:18:21 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:18:21 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:18:21 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:18:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:47006->10.242.244.80:8301: i/o timeout
2021/04/17 11:18:21 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:18:21 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:18:21 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:18:22 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500393373s
2021/04/17 11:18:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:18:24 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.979442534s
2021/04/17 11:18:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:18:26 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:18:27 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:18:28 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:18:29 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:18:29 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:18:29 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:18:31 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:18:31 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:18:32 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500259734s
2021/04/17 11:18:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:47180->10.242.244.80:8301: i/o timeout
2021/04/17 11:18:32 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:18:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:18:34 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.993437092s
2021/04/17 11:18:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:18:36 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:18:36 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:18:36 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:18:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:47238->10.242.244.80:8301: i/o timeout
2021/04/17 11:18:36 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:18:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:18:36 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 7.450944295s
2021/04/17 11:18:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:18:39 [INFO] raft: aborting pipeline replication to peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}
2021/04/17 11:18:39 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:18:40 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:33732->10.242.244.80:8300: i/o timeout
2021/04/17 11:18:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:18:40 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:18:40 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:18:41 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:18:42 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500312296s
2021/04/17 11:18:43 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:18:43 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:18:44 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.959926932s
2021/04/17 11:18:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:18:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:18:46 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 7.439540243s
2021/04/17 11:18:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:34144->10.242.244.80:8302: i/o timeout
2021/04/17 11:18:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:18:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:18:49 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:35716->10.242.244.80:8300: i/o timeout
2021/04/17 11:18:49 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:18:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:18:51 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:18:51 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:38500
2021/04/17 11:18:51 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:35758->10.242.244.80:8300: i/o timeout
2021/04/17 11:18:51 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:18:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:18:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:18:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:18:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:34222->10.242.244.80:8302: i/o timeout
2021/04/17 11:18:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:18:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:18:59 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:35874->10.242.244.80:8300: i/o timeout
2021/04/17 11:18:59 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:19:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:01 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:35914->10.242.244.80:8300: i/o timeout
2021/04/17 11:19:02 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:19:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:09 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:36000->10.242.244.80:8300: i/o timeout
2021/04/17 11:19:09 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:19:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:12 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:19:12 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:19:12 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:19:12 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:36044->10.242.244.80:8300: i/o timeout
2021/04/17 11:19:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:13 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:19:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:18 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:19:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:19 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:36150->10.242.244.80:8300: i/o timeout
2021/04/17 11:19:19 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:19:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:22 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:19:22 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:19:23 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:36206->10.242.244.80:8300: i/o timeout
2021/04/17 11:19:24 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:19:25 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:19:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:29 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:36288->10.242.244.80:8300: i/o timeout
2021/04/17 11:19:29 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:19:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:34 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:36344->10.242.244.80:8300: i/o timeout
2021/04/17 11:19:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:35 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:19:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:39 [ERR] memberlist: failed to receive: read tcp 10.242.146.73:8301->10.242.244.80:49946: i/o timeout from=10.242.244.80:49946
2021/04/17 11:19:39 [ERR] memberlist: Failed to send error: write tcp 10.242.146.73:8301->10.242.244.80:49946: i/o timeout from=10.242.244.80:49946
2021/04/17 11:19:39 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:36414->10.242.244.80:8300: i/o timeout
2021/04/17 11:19:39 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:19:39 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:19:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:45 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:36482->10.242.244.80:8300: i/o timeout
2021/04/17 11:19:45 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:19:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:49 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:36548->10.242.244.80:8300: i/o timeout
2021/04/17 11:19:50 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:19:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:55 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:36622->10.242.244.80:8300: i/o timeout
2021/04/17 11:19:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:19:57 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:19:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:00 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:36676->10.242.244.80:8300: i/o timeout
2021/04/17 11:20:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:00 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:20:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:04 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:20:04 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:20:07 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:36774->10.242.244.80:8300: i/o timeout
2021/04/17 11:20:07 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:20:07 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:20:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:09 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:20:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:10 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:36828->10.242.244.80:8300: i/o timeout
2021/04/17 11:20:12 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:20:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:19 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:36936->10.242.244.80:8300: i/o timeout
2021/04/17 11:20:20 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:20:20 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:20:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:22 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:36996->10.242.244.80:8300: i/o timeout
2021/04/17 11:20:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:22 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:20:24 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:20:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:35562->10.242.244.80:8302: i/o timeout
2021/04/17 11:20:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:20:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:32 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:37132->10.242.244.80:8300: i/o timeout
2021/04/17 11:20:34 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:37158->10.242.244.80:8300: i/o timeout
2021/04/17 11:20:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:38 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:20:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:39 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:20:40 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:20:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:48 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:37446->10.242.244.80:8300: i/o timeout
2021/04/17 11:20:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:49 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:37454->10.242.244.80:8300: i/o timeout
2021/04/17 11:20:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:35954->10.242.244.80:8302: i/o timeout
2021/04/17 11:20:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:20:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:57 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:20:57 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:20:57 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:20:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:20:59 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:21:00 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:21:00 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:21:00 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:41244
2021/04/17 11:21:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:36022->10.242.244.80:8302: i/o timeout
2021/04/17 11:21:02 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:21:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:09 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:37744->10.242.244.80:8300: i/o timeout
2021/04/17 11:21:10 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:37748->10.242.244.80:8300: i/o timeout
2021/04/17 11:21:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:15 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:21:15 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:21:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:20 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:21:20 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:21:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:36350->10.242.244.80:8302: i/o timeout
2021/04/17 11:21:22 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:21:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:30 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:38014->10.242.244.80:8300: i/o timeout
2021/04/17 11:21:30 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:38026->10.242.244.80:8300: i/o timeout
2021/04/17 11:21:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:36550->10.242.244.80:8302: i/o timeout
2021/04/17 11:21:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:21:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:40 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:21:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:41 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:21:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:36618->10.242.244.80:8302: i/o timeout
2021/04/17 11:21:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:21:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:50 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:38286->10.242.244.80:8300: i/o timeout
2021/04/17 11:21:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:51 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:21:51 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:21:51 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:38318->10.242.244.80:8300: i/o timeout
2021/04/17 11:21:52 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:21:52 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:21:52 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:21:53 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:21:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:21:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:36820->10.242.244.80:8302: i/o timeout
2021/04/17 11:21:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:21:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:00 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:22:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:02 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:22:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:10 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:38576->10.242.244.80:8300: i/o timeout
2021/04/17 11:22:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:12 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:38598->10.242.244.80:8300: i/o timeout
2021/04/17 11:22:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:21 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:22:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:23 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:22:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:31 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:38836->10.242.244.80:8300: i/o timeout
2021/04/17 11:22:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:32 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:22:32 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:22:33 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:38868->10.242.244.80:8300: i/o timeout
2021/04/17 11:22:35 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:22:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:39 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:22:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:41 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:22:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:44 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:22:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:51 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:39090->10.242.244.80:8300: i/o timeout
2021/04/17 11:22:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:54 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:39140->10.242.244.80:8300: i/o timeout
2021/04/17 11:22:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:22:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:01 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:23:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:05 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:23:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:11 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:39372->10.242.244.80:8300: i/o timeout
2021/04/17 11:23:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:14 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:23:14 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:23:15 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:39430->10.242.244.80:8300: i/o timeout
2021/04/17 11:23:17 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:23:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:21 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:23:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:26 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:23:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:28 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:23:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:31 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:39656->10.242.244.80:8300: i/o timeout
2021/04/17 11:23:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:36 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:39720->10.242.244.80:8300: i/o timeout
2021/04/17 11:23:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:42 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:23:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:47 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:23:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:52 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:40008->10.242.244.80:8300: i/o timeout
2021/04/17 11:23:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:56 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:23:56 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:23:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:23:56 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:23:56 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:23:57 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:40078->10.242.244.80:8300: i/o timeout
2021/04/17 11:23:59 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:24:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:02 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:24:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:08 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:24:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:12 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:40266->10.242.244.80:8300: i/o timeout
2021/04/17 11:24:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:38774->10.242.244.80:8302: i/o timeout
2021/04/17 11:24:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:24:18 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:40358->10.242.244.80:8300: i/o timeout
2021/04/17 11:24:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:22 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:24:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:38832->10.242.244.80:8302: i/o timeout
2021/04/17 11:24:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:24:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:29 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:24:30 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:24:30 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:41936
2021/04/17 11:24:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:32 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:40526->10.242.244.80:8300: i/o timeout
2021/04/17 11:24:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:38 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:24:38 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:24:39 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:40618->10.242.244.80:8300: i/o timeout
2021/04/17 11:24:41 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:24:41 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 11:24:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:39132->10.242.244.80:8302: i/o timeout
2021/04/17 11:24:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:24:42 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:24:42 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:24:42 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:24:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:50 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:24:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:52 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:40834->10.242.244.80:8300: i/o timeout
2021/04/17 11:24:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:24:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:00 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:40956->10.242.244.80:8300: i/o timeout
2021/04/17 11:25:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:02 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:25:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:05 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:25:05 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:25:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:08 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:25:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:10 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:25:10 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:25:10 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:25:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:52844->10.242.244.80:8301: i/o timeout
2021/04/17 11:25:12 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:25:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:39586->10.242.244.80:8302: i/o timeout
2021/04/17 11:25:12 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:25:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:13 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:41120->10.242.244.80:8300: i/o timeout
2021/04/17 11:25:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:15 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:25:15 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:25:15 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:25:15 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.146.73:8300
2021/04/17 11:25:15 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/17 11:25:15 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:25:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:17 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500292136s
2021/04/17 11:25:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:19 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.994762359s
2021/04/17 11:25:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:22 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 7.482236611s
2021/04/17 11:25:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:23 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:25:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:25 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:41288->10.242.244.80:8300: i/o timeout
2021/04/17 11:25:26 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:25:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:39744->10.242.244.80:8302: i/o timeout
2021/04/17 11:25:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:25:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:31 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:25:31 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:25:31 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:25:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:33 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:41444->10.242.244.80:8300: i/o timeout
2021/04/17 11:25:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:53156->10.242.244.80:8301: i/o timeout
2021/04/17 11:25:33 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:25:34 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:25:34 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:42472
2021/04/17 11:25:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:36 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:41508->10.242.244.80:8300: i/o timeout
2021/04/17 11:25:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:53198->10.242.244.80:8301: i/o timeout
2021/04/17 11:25:36 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:25:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:37 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:25:37 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:25:37 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:25:37 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:25:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:39964->10.242.244.80:8302: i/o timeout
2021/04/17 11:25:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:25:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:53210->10.242.244.80:8301: i/o timeout
2021/04/17 11:25:39 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:25:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:42 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:25:42 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:25:42 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:25:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:43 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:25:44 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:25:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:47 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:41656->10.242.244.80:8300: i/o timeout
2021/04/17 11:25:47 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:25:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:53 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:41744->10.242.244.80:8300: i/o timeout
2021/04/17 11:25:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:57 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:41812->10.242.244.80:8300: i/o timeout
2021/04/17 11:25:58 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:25:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:25:59 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:25:59 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:25:59 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:26:00 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:26:00 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:26:03 [INFO] raft: pipelining replication to peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}
2021/04/17 11:26:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:53968->10.242.244.80:8301: i/o timeout
2021/04/17 11:26:23 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:26:23 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.506256743s
2021/04/17 11:26:31 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:27:01 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:27:11 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:27:12 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500141676s
2021/04/17 11:27:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:54990->10.242.244.80:8301: i/o timeout
2021/04/17 11:27:12 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:27:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:27:14 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:27:14 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:27:14 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:27:14 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.972449614s
2021/04/17 11:27:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:27:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:27:17 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 7.443774201s
2021/04/17 11:27:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:41850->10.242.244.80:8302: i/o timeout
2021/04/17 11:27:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:27:18 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:27:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:27:19 [INFO] raft: aborting pipeline replication to peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}
2021/04/17 11:27:19 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:27:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:27:21 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.50017887s
2021/04/17 11:27:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:27:24 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.99585159s
2021/04/17 11:27:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:27:25 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:27:25 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:27:26 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 7.447576644s
2021/04/17 11:27:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:41944->10.242.244.80:8302: i/o timeout
2021/04/17 11:27:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:27:27 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:27:28 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:42064->10.242.244.80:8300: i/o timeout
2021/04/17 11:27:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:27:28 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:27:29 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:43596->10.242.244.80:8300: i/o timeout
2021/04/17 11:27:29 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:27:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:27:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:27:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:27:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:27:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:27:39 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:43776->10.242.244.80:8300: i/o timeout
2021/04/17 11:27:39 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:43782->10.242.244.80:8300: i/o timeout
2021/04/17 11:27:39 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:27:39 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:27:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:27:41 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:27:41 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:43316
2021/04/17 11:27:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:42304->10.242.244.80:8302: i/o timeout
2021/04/17 11:27:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:27:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:27:44 [ERR] memberlist: failed to receive: read tcp 10.242.146.73:8301->10.242.244.80:53926: i/o timeout from=10.242.244.80:53926
2021/04/17 11:27:44 [ERR] memberlist: Failed to send error: write tcp 10.242.146.73:8301->10.242.244.80:53926: i/o timeout from=10.242.244.80:53926
2021/04/17 11:27:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:27:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:27:47 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:27:47 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:27:47 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:27:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:27:49 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:43950->10.242.244.80:8300: i/o timeout
2021/04/17 11:27:49 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:27:49 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:43952->10.242.244.80:8300: i/o timeout
2021/04/17 11:27:50 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:27:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:27:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:27:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:27:54 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:27:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:27:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:27:59 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:44120->10.242.244.80:8300: i/o timeout
2021/04/17 11:27:59 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:28:00 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:44132->10.242.244.80:8300: i/o timeout
2021/04/17 11:28:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:28:01 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:28:02 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:28:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:28:03 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:28:03 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:28:03 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:28:03 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:28:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:28:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:55936->10.242.244.80:8301: i/o timeout
2021/04/17 11:28:06 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:28:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:28:06 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:28:06 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:28:09 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:28:09 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:28:09 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:28:09 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:28:09 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:44286->10.242.244.80:8300: i/o timeout
2021/04/17 11:28:09 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:28:09 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:28:09 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:28:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:28:11 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:44324->10.242.244.80:8300: i/o timeout
2021/04/17 11:28:12 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:28:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:28:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:28:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:28:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:28:19 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:28:19 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:28:19 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:44452->10.242.244.80:8300: i/o timeout
2021/04/17 11:28:19 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:28:19 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:28:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:28:21 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500293728s
2021/04/17 11:28:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:28:23 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.958863517s
2021/04/17 11:28:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:28:24 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:28:25 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:28:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:28:27 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:28:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:43048->10.242.244.80:8302: i/o timeout
2021/04/17 11:28:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:28:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:28:29 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500256823s
2021/04/17 11:28:29 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:44624->10.242.244.80:8300: i/o timeout
2021/04/17 11:28:30 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:28:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:28:31 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.961808193s
2021/04/17 11:28:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:28:33 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:28:34 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:28:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:28:35 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:28:36 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:28:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:28:37 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:28:37 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:28:38 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:28:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:28:39 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:28:39 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:28:40 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:44784->10.242.244.80:8300: i/o timeout
2021/04/17 11:28:40 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:28:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:28:41 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500269218s
2021/04/17 11:28:42 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.146.73:8300
2021/04/17 11:28:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:43278->10.242.244.80:8302: i/o timeout
2021/04/17 11:28:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:28:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:28:44 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.953164631s
2021/04/17 11:28:45 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:28:46 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 7.396492857s
2021/04/17 11:28:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:28:47 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:28:47 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:28:47 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:28:47 [ERR] raft: peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300} has newer term, stopping replication
2021/04/17 11:28:47 [INFO] raft: Node at 10.242.146.73:8300 [Follower] entering Follower state (Leader: "")
2021/04/17 11:28:47 [INFO] raft: aborting pipeline replication to peer {Voter e3d1abd7-c4ef-d7bc-57e1-b271f273a71b 10.242.253.198:8300}
2021/04/17 11:28:47 [INFO] consul: cluster leadership lost
2021/04/17 11:28:47 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since our last index is greater (709, 685)
2021/04/17 11:28:49 [WARN] consul.coordinate: Batch update failed: node is not the leader
2021/04/17 11:28:53 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/17 11:28:53 [INFO] raft: Node at 10.242.146.73:8300 [Candidate] entering Candidate state in term 14
2021/04/17 11:28:53 [INFO] raft: Election won. Tally: 2
2021/04/17 11:28:53 [INFO] raft: Node at 10.242.146.73:8300 [Leader] entering Leader state
2021/04/17 11:28:53 [INFO] raft: Added peer 79007099-56a4-82c8-2a6b-8367b805feea, starting replication
2021/04/17 11:28:53 [INFO] raft: Added peer e3d1abd7-c4ef-d7bc-57e1-b271f273a71b, starting replication
2021/04/17 11:28:53 [INFO] consul: cluster leadership acquired
2021/04/17 11:28:53 [INFO] consul: New leader elected: dev-consul-server-1
2021/04/17 11:28:53 [INFO] raft: pipelining replication to peer {Voter e3d1abd7-c4ef-d7bc-57e1-b271f273a71b 10.242.253.198:8300}
2021/04/17 11:28:54 [WARN] raft: AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300} rejected, sending older logs (next: 686)
2021/04/17 11:28:54 [INFO] raft: pipelining replication to peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}
2021/04/17 11:28:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:28:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:29:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:56776->10.242.244.80:8301: i/o timeout
2021/04/17 11:29:00 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:29:04 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:29:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:29:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:56874->10.242.244.80:8301: i/o timeout
2021/04/17 11:29:06 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:29:06 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500244092s
2021/04/17 11:29:07 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:29:16 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:29:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:57042->10.242.244.80:8301: i/o timeout
2021/04/17 11:29:18 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:29:20 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:29:40 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:30:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:57818->10.242.244.80:8301: i/o timeout
2021/04/17 11:30:16 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:30:24 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:30:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:30:26 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500241806s
2021/04/17 11:30:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:58144->10.242.244.80:8301: i/o timeout
2021/04/17 11:30:27 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:30:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:30:29 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:30:29 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:30:29 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:30:29 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:30:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:30:30 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500240331s
2021/04/17 11:30:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:30:32 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.980005541s
2021/04/17 11:30:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:30:34 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 7.402343643s
2021/04/17 11:30:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:30:37 [INFO] raft: aborting pipeline replication to peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}
2021/04/17 11:30:37 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:30:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:45032->10.242.244.80:8302: i/o timeout
2021/04/17 11:30:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:30:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:30:38 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:44512->10.242.244.80:8300: i/o timeout
2021/04/17 11:30:39 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:30:39 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500332189s
2021/04/17 11:30:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:30:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:30:41 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.926972795s
2021/04/17 11:30:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:30:44 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 7.400618603s
2021/04/17 11:30:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:30:47 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:46722->10.242.244.80:8300: i/o timeout
2021/04/17 11:30:47 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:30:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:45182->10.242.244.80:8302: i/o timeout
2021/04/17 11:30:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:30:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:30:49 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:46746->10.242.244.80:8300: i/o timeout
2021/04/17 11:30:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:30:49 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:30:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:30:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:30:54 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:30:54 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:30:56 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:30:57 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:46872->10.242.244.80:8300: i/o timeout
2021/04/17 11:30:57 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:30:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:30:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:30:59 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:46918->10.242.244.80:8300: i/o timeout
2021/04/17 11:31:00 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:31:00 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:45292
2021/04/17 11:31:00 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:31:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:45400->10.242.244.80:8302: i/o timeout
2021/04/17 11:31:02 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:31:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:04 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:31:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:07 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:47022->10.242.244.80:8300: i/o timeout
2021/04/17 11:31:07 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:31:07 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:31:07 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:31:07 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:31:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:10 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:47072->10.242.244.80:8300: i/o timeout
2021/04/17 11:31:11 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:31:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:45474->10.242.244.80:8302: i/o timeout
2021/04/17 11:31:12 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:31:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:17 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.146.73:8300
2021/04/17 11:31:17 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:47170->10.242.244.80:8300: i/o timeout
2021/04/17 11:31:17 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:31:17 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:31:17 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:31:17 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:31:18 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:31:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:58908->10.242.244.80:8301: i/o timeout
2021/04/17 11:31:20 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:31:21 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:47242->10.242.244.80:8300: i/o timeout
2021/04/17 11:31:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:23 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:31:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:58950->10.242.244.80:8301: i/o timeout
2021/04/17 11:31:23 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:31:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:24 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:31:24 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:31:24 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:31:24 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:31:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:27 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:47314->10.242.244.80:8300: i/o timeout
2021/04/17 11:31:27 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:31:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:32 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:47400->10.242.244.80:8300: i/o timeout
2021/04/17 11:31:33 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:31:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:35 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:31:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:35 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:31:37 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:47474->10.242.244.80:8300: i/o timeout
2021/04/17 11:31:37 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:31:38 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:31:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:43 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:47568->10.242.244.80:8300: i/o timeout
2021/04/17 11:31:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:44 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:31:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:47 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:47624->10.242.244.80:8300: i/o timeout
2021/04/17 11:31:47 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:31:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:48 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:31:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:50 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.503115581s
2021/04/17 11:31:50 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.146.73:8300
2021/04/17 11:31:51 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:31:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:31:51 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:31:51 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:31:51 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:31:51 [ERR] raft: peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300} has newer term, stopping replication
2021/04/17 11:31:51 [INFO] raft: Node at 10.242.146.73:8300 [Follower] entering Follower state (Leader: "")
2021/04/17 11:31:51 [INFO] raft: aborting pipeline replication to peer {Voter e3d1abd7-c4ef-d7bc-57e1-b271f273a71b 10.242.253.198:8300}
2021/04/17 11:31:51 [INFO] consul: cluster leadership lost
2021/04/17 11:31:51 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:31:51 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:31:59 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.146.75:34980
2021/04/17 11:31:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:59496->10.242.244.80:8301: i/o timeout
2021/04/17 11:31:59 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:32:00 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/17 11:32:00 [INFO] raft: Node at 10.242.146.73:8300 [Candidate] entering Candidate state in term 15
2021/04/17 11:32:00 [INFO] raft: Election won. Tally: 2
2021/04/17 11:32:00 [INFO] raft: Node at 10.242.146.73:8300 [Leader] entering Leader state
2021/04/17 11:32:00 [INFO] raft: Added peer 79007099-56a4-82c8-2a6b-8367b805feea, starting replication
2021/04/17 11:32:00 [INFO] raft: Added peer e3d1abd7-c4ef-d7bc-57e1-b271f273a71b, starting replication
2021/04/17 11:32:00 [INFO] consul: cluster leadership acquired
2021/04/17 11:32:00 [INFO] consul: New leader elected: dev-consul-server-1
2021/04/17 11:32:00 [INFO] raft: pipelining replication to peer {Voter e3d1abd7-c4ef-d7bc-57e1-b271f273a71b 10.242.253.198:8300}
2021/04/17 11:32:01 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:32:01 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:32:01 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:32:01 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:32:01 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:32:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:46300->10.242.244.80:8302: i/o timeout
2021/04/17 11:32:02 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:32:03 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.503114838s
2021/04/17 11:32:03 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:32:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:05 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.98331837s
2021/04/17 11:32:06 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:32:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:06 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:32:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:08 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500797364s
2021/04/17 11:32:09 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:32:09 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:32:09 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 11:32:10 [ERR] raft: Failed to make RequestVote RPC to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:47738->10.242.244.80:8300: i/o timeout
2021/04/17 11:32:10 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:47796->10.242.244.80:8300: i/o timeout
2021/04/17 11:32:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:59678->10.242.244.80:8301: i/o timeout
2021/04/17 11:32:11 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:32:11 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.940250779s
2021/04/17 11:32:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:12 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.146.73:8300
2021/04/17 11:32:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:59712->10.242.244.80:8301: i/o timeout
2021/04/17 11:32:14 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:32:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:15 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:32:15 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:32:15 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:32:15 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 11:32:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:59758->10.242.244.80:8301: i/o timeout
2021/04/17 11:32:16 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:32:16 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 2.500360485s
2021/04/17 11:32:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:18 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 4.964913517s
2021/04/17 11:32:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:20 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:48110->10.242.244.80:8300: i/o timeout
2021/04/17 11:32:20 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:32:21 [WARN] raft: Failed to contact 79007099-56a4-82c8-2a6b-8367b805feea in 7.448497604s
2021/04/17 11:32:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:46596->10.242.244.80:8302: i/o timeout
2021/04/17 11:32:22 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:32:23 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:32:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:24 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:47984->10.242.244.80:8300: i/o timeout
2021/04/17 11:32:25 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:32:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:30 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:48270->10.242.244.80:8300: i/o timeout
2021/04/17 11:32:30 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:32:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:46750->10.242.244.80:8302: i/o timeout
2021/04/17 11:32:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:32:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:35 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:48336->10.242.244.80:8300: i/o timeout
2021/04/17 11:32:36 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:32:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:40 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:48418->10.242.244.80:8300: i/o timeout
2021/04/17 11:32:40 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:32:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:47006->10.242.244.80:8302: i/o timeout
2021/04/17 11:32:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:32:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:44 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:32:46 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:48522->10.242.244.80:8300: i/o timeout
2021/04/17 11:32:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:47 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:32:47 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:32:47 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:32:47 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:32:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:50 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:48674->10.242.244.80:8300: i/o timeout
2021/04/17 11:32:50 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:32:51 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:32:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:32:57 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:48776->10.242.244.80:8300: i/o timeout
2021/04/17 11:32:57 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:32:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:00 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:48830->10.242.244.80:8300: i/o timeout
2021/04/17 11:33:00 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:33:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:07 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:48932->10.242.244.80:8300: i/o timeout
2021/04/17 11:33:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:08 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:33:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:10 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:48972->10.242.244.80:8300: i/o timeout
2021/04/17 11:33:11 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:33:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:18 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:49096->10.242.244.80:8300: i/o timeout
2021/04/17 11:33:19 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:33:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:21 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:49130->10.242.244.80:8300: i/o timeout
2021/04/17 11:33:21 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:33:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:24 [ERR] memberlist: failed to receive: read tcp 10.242.146.73:8302->10.242.244.80:39568: i/o timeout from=10.242.244.80:39568
2021/04/17 11:33:24 [ERR] memberlist: Failed to send error: write tcp 10.242.146.73:8302->10.242.244.80:39568: i/o timeout from=10.242.244.80:39568
2021/04/17 11:33:24 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:33:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:29 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:33:29 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:33:29 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:49258->10.242.244.80:8300: i/o timeout
2021/04/17 11:33:30 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:33:31 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:33:31 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:49318->10.242.244.80:8300: i/o timeout
2021/04/17 11:33:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:33 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:33:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:40 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:49452->10.242.244.80:8300: i/o timeout
2021/04/17 11:33:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:41 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:33:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:43 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:49506->10.242.244.80:8300: i/o timeout
2021/04/17 11:33:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:45 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:33:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:51 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:49646->10.242.244.80:8300: i/o timeout
2021/04/17 11:33:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:53 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:33:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:55 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:49700->10.242.244.80:8300: i/o timeout
2021/04/17 11:33:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:33:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:34:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:34:00 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:34:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:34:03 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:49820->10.242.244.80:8300: i/o timeout
2021/04/17 11:34:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:34:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:34:07 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:34:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:34:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:34:10 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:34:10 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 11:34:10 [ERR] raft: Failed to AppendEntries to {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300}: read tcp 10.242.146.73:49914->10.242.244.80:8300: i/o timeout
2021/04/17 11:34:13 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:34:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:34:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:34:16 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:34:16 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:34:17 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:50012->10.242.244.80:8300: i/o timeout
2021/04/17 11:34:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:34:19 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.146.73:8300
2021/04/17 11:34:21 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:34:21 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 11:34:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 11:34:22 [WARN] Unable to get address for server id 79007099-56a4-82c8-2a6b-8367b805feea, using fallback address 10.242.244.80:8300: Could not find address for server id 79007099-56a4-82c8-2a6b-8367b805feea
2021/04/17 11:34:23 [ERR] raft: peer {Voter 79007099-56a4-82c8-2a6b-8367b805feea 10.242.244.80:8300} has newer term, stopping replication
2021/04/17 11:34:23 [INFO] raft: Node at 10.242.146.73:8300 [Follower] entering Follower state (Leader: "")
2021/04/17 11:34:23 [INFO] raft: aborting pipeline replication to peer {Voter e3d1abd7-c4ef-d7bc-57e1-b271f273a71b 10.242.253.198:8300}
2021/04/17 11:34:23 [INFO] consul: cluster leadership lost
2021/04/17 11:34:23 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:34:23 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:34:28 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 11:34:28 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/17 11:34:31 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:34:32 [INFO] serf: EventMemberUpdate: dev-consul-server-2
2021/04/17 11:34:32 [INFO] serf: EventMemberUpdate: dev-consul-server-2.dc1
2021/04/17 11:34:34 [DEBUG] raft-net: 10.242.146.73:8300 accepted connection from: 10.242.253.198:47640
2021/04/17 11:34:35 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/17 11:34:35 [DEBUG] raft-net: 10.242.146.73:8300 accepted connection from: 10.242.253.198:47656
2021/04/17 11:34:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:33938->10.242.244.80:8301: i/o timeout
2021/04/17 11:34:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:33986->10.242.244.80:8301: i/o timeout
2021/04/17 11:34:49 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:34:51 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:34:51 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:34:51 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:34:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:33998->10.242.244.80:8301: i/o timeout
2021/04/17 11:34:51 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:34:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:49018->10.242.244.80:8302: i/o timeout
2021/04/17 11:34:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:35:01 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:35:01 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:35:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:34830->10.242.244.80:8301: i/o timeout
2021/04/17 11:35:43 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:35:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:34844->10.242.244.80:8301: i/o timeout
2021/04/17 11:35:46 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:35:47 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:35:47 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:35:47 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:35:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:49956->10.242.244.80:8302: i/o timeout
2021/04/17 11:35:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:36:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:50092->10.242.244.80:8302: i/o timeout
2021/04/17 11:36:02 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:36:12 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:36:12 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:49864
2021/04/17 11:36:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:50380->10.242.244.80:8302: i/o timeout
2021/04/17 11:36:22 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:36:22 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:36:22 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:36:22 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:36:33 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:36:33 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:36:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:35602->10.242.244.80:8301: i/o timeout
2021/04/17 11:36:36 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:36:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:35646->10.242.244.80:8301: i/o timeout
2021/04/17 11:36:40 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:36:40 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:36:40 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:36:44 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:38:27 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:38:45 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:38:45 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:38:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:37680->10.242.244.80:8301: i/o timeout
2021/04/17 11:38:51 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:38:51 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:38:51 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:38:51 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:39:04 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:39:04 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:39:04 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:39:04 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:39:06 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/17 11:39:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:38024->10.242.244.80:8301: i/o timeout
2021/04/17 11:39:14 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:39:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:38088->10.242.244.80:8301: i/o timeout
2021/04/17 11:39:17 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:39:18 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:39:18 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:39:18 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:39:18 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:39:18 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:39:20 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:39:24 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:39:24 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:39:24 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:39:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:39:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:39:51 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:39:51 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:39:51 [INFO] serf: EventMemberUpdate: dev-consul-server-2.dc1
2021/04/17 11:40:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:38992->10.242.244.80:8301: i/o timeout
2021/04/17 11:40:19 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:40:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:39000->10.242.244.80:8301: i/o timeout
2021/04/17 11:40:21 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:40:23 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:40:23 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:40:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:54156->10.242.244.80:8302: i/o timeout
2021/04/17 11:40:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:40:33 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:40:33 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:40:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:39242->10.242.244.80:8301: i/o timeout
2021/04/17 11:40:37 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:40:38 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:40:38 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:40:38 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:40:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:39256->10.242.244.80:8301: i/o timeout
2021/04/17 11:40:40 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:40:46 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:40:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:54296->10.242.244.80:8302: i/o timeout
2021/04/17 11:40:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:41:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:54596->10.242.244.80:8302: i/o timeout
2021/04/17 11:41:02 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:41:12 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/17 11:41:17 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:41:17 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:41:17 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:41:18 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:41:18 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:41:19 [DEBUG] raft-net: 10.242.146.73:8300 accepted connection from: 10.242.244.80:54762
2021/04/17 11:41:19 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:41:19 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 11:41:19 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:41:19 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:41:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:40120->10.242.244.80:8301: i/o timeout
2021/04/17 11:41:35 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:41:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:40142->10.242.244.80:8301: i/o timeout
2021/04/17 11:41:37 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:41:39 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:41:39 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:41:39 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:41:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:55316->10.242.244.80:8302: i/o timeout
2021/04/17 11:41:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:41:48 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:41:48 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:41:48 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:41:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:40486->10.242.244.80:8301: i/o timeout
2021/04/17 11:41:52 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:41:58 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/17 11:41:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:40582->10.242.244.80:8301: i/o timeout
2021/04/17 11:41:59 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:42:01 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:42:01 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:42:01 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:42:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:40632->10.242.244.80:8301: i/o timeout
2021/04/17 11:42:02 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:42:04 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:42:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:55688->10.242.244.80:8302: i/o timeout
2021/04/17 11:42:07 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:42:11 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:42:11 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:42:11 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:42:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:40788->10.242.244.80:8301: i/o timeout
2021/04/17 11:42:14 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:42:16 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:42:16 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:42:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:40834->10.242.244.80:8301: i/o timeout
2021/04/17 11:42:16 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:42:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:55978->10.242.244.80:8302: i/o timeout
2021/04/17 11:42:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:42:33 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:42:33 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:42:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:41118->10.242.244.80:8301: i/o timeout
2021/04/17 11:42:35 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:42:37 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/17 11:42:39 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:42:39 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:42:39 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:42:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:41176->10.242.244.80:8301: i/o timeout
2021/04/17 11:42:39 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:42:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:56278->10.242.244.80:8302: i/o timeout
2021/04/17 11:42:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:42:52 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:42:52 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:42:52 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:43:11 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:43:12 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:43:12 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:43:13 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:43:13 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:43:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:41710->10.242.244.80:8301: i/o timeout
2021/04/17 11:43:15 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:43:19 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:43:19 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:43:19 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:43:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:41778->10.242.244.80:8301: i/o timeout
2021/04/17 11:43:19 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:43:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:56806->10.242.244.80:8302: i/o timeout
2021/04/17 11:43:22 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:43:22 [ERR] memberlist: failed to receive: read tcp 10.242.146.73:8301->10.242.244.80:37124: i/o timeout from=10.242.244.80:37124
2021/04/17 11:43:22 [ERR] memberlist: Failed to send error: write tcp 10.242.146.73:8301->10.242.244.80:37124: i/o timeout from=10.242.244.80:37124
2021/04/17 11:43:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:56960->10.242.244.80:8302: i/o timeout
2021/04/17 11:43:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:43:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:57034->10.242.244.80:8302: i/o timeout
2021/04/17 11:43:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:43:51 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:43:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:57260->10.242.244.80:8302: i/o timeout
2021/04/17 11:43:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:43:52 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:43:52 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:43:52 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:43:57 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:44:18 [ERR] memberlist: failed to receive: read tcp 10.242.146.73:8302->10.242.244.80:47530: i/o timeout from=10.242.244.80:47530
2021/04/17 11:44:18 [ERR] memberlist: Failed to send error: write tcp 10.242.146.73:8302->10.242.244.80:47530: i/o timeout from=10.242.244.80:47530
2021/04/17 11:44:19 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:44:19 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:44:19 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:44:19 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.253.198
2021/04/17 11:44:19 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.253.198:8300) (DC: dc1)
2021/04/17 11:44:19 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.253.198
2021/04/17 11:44:19 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.253.198:8300) (DC: dc1)
2021/04/17 11:44:20 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 11:44:31 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:44:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:57804->10.242.244.80:8302: i/o timeout
2021/04/17 11:44:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:44:32 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:44:32 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:44:39 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/17 11:44:40 [DEBUG] raft-net: 10.242.146.73:8300 accepted connection from: 10.242.253.198:59470
2021/04/17 11:49:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:47706->10.242.244.80:8301: i/o timeout
2021/04/17 11:49:59 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:50:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:47856->10.242.244.80:8301: i/o timeout
2021/04/17 11:50:09 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:50:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:34652->10.242.244.80:8302: i/o timeout
2021/04/17 11:50:13 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:50:13 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:50:13 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:50:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:34806->10.242.244.80:8302: i/o timeout
2021/04/17 11:50:22 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:50:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:34952->10.242.244.80:8302: i/o timeout
2021/04/17 11:50:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:50:38 [ERR] memberlist: Push/Pull with dev-consul-server-2.dc1 failed: read tcp 10.242.146.73:34624->10.242.244.80:8302: i/o timeout
2021/04/17 11:50:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:35038->10.242.244.80:8302: i/o timeout
2021/04/17 11:50:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:50:51 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 11:50:51 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:60894
2021/04/17 11:50:52 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:50:52 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:50:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:35372->10.242.244.80:8302: i/o timeout
2021/04/17 11:50:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:50:52 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:50:57 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:50:57 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:50:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:48720->10.242.244.80:8301: i/o timeout
2021/04/17 11:50:59 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:51:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:48768->10.242.244.80:8301: i/o timeout
2021/04/17 11:51:02 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:51:03 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:51:03 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:51:03 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:51:11 [ERR] memberlist: failed to receive: read tcp 10.242.146.73:8301->10.242.244.80:43160: i/o timeout from=10.242.244.80:43160
2021/04/17 11:51:11 [ERR] memberlist: Failed to send error: write tcp 10.242.146.73:8301->10.242.244.80:43160: i/o timeout from=10.242.244.80:43160
2021/04/17 11:51:24 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:51:24 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:51:24 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:51:24 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 11:51:24 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:51:24 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:51:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:49146->10.242.244.80:8301: i/o timeout
2021/04/17 11:51:28 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:51:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:49208->10.242.244.80:8301: i/o timeout
2021/04/17 11:51:32 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:51:35 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:51:35 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:51:35 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:51:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:49246->10.242.244.80:8301: i/o timeout
2021/04/17 11:51:35 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:51:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:36116->10.242.244.80:8302: i/o timeout
2021/04/17 11:51:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:51:46 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:51:46 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:51:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:49456->10.242.244.80:8301: i/o timeout
2021/04/17 11:51:48 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:51:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:36270->10.242.244.80:8302: i/o timeout
2021/04/17 11:51:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:51:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:49522->10.242.244.80:8301: i/o timeout
2021/04/17 11:51:52 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:51:52 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:51:52 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:51:52 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:51:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:49532->10.242.244.80:8301: i/o timeout
2021/04/17 11:51:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:52:06 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 11:52:07 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:52:07 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:52:07 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:52:07 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:52:09 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/17 11:52:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:49800->10.242.244.80:8301: i/o timeout
2021/04/17 11:52:11 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:52:13 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:52:13 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:52:13 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:52:14 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:52:14 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:52:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:49912->10.242.244.80:8301: i/o timeout
2021/04/17 11:52:18 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:52:21 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/17 11:52:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:49982->10.242.244.80:8301: i/o timeout
2021/04/17 11:52:23 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:52:26 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/17 11:52:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:50072->10.242.244.80:8301: i/o timeout
2021/04/17 11:52:29 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:52:30 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 11:52:31 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:52:31 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:52:31 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:52:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:37018->10.242.244.80:8302: i/o timeout
2021/04/17 11:52:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:52:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:37162->10.242.244.80:8302: i/o timeout
2021/04/17 11:52:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:37236->10.242.244.80:8302: i/o timeout
2021/04/17 11:52:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:53:05 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:53:05 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:53:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:51044->10.242.244.80:8301: i/o timeout
2021/04/17 11:53:36 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:53:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:51294->10.242.244.80:8301: i/o timeout
2021/04/17 11:53:47 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:53:48 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:53:48 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:53:48 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:53:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:38188->10.242.244.80:8302: i/o timeout
2021/04/17 11:53:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:54:02 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:54:03 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:54:03 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:54:03 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:54:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:51592->10.242.244.80:8301: i/o timeout
2021/04/17 11:54:07 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:54:08 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:54:08 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:54:08 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:54:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:51622->10.242.244.80:8301: i/o timeout
2021/04/17 11:54:10 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:54:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:38420->10.242.244.80:8302: i/o timeout
2021/04/17 11:54:12 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:54:18 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:54:18 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:54:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:51776->10.242.244.80:8301: i/o timeout
2021/04/17 11:54:19 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:54:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:51818->10.242.244.80:8301: i/o timeout
2021/04/17 11:54:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:54:23 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:54:23 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:54:23 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:54:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:51826->10.242.244.80:8301: i/o timeout
2021/04/17 11:54:24 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:54:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:38704->10.242.244.80:8302: i/o timeout
2021/04/17 11:54:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:54:35 [ERR] memberlist: Push/Pull with dev-consul-server-2.dc1 failed: read tcp 10.242.146.73:38340->10.242.244.80:8302: i/o timeout
2021/04/17 11:54:42 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:54:42 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:54:42 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:54:52 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:54:52 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:55:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:39124->10.242.244.80:8302: i/o timeout
2021/04/17 11:55:02 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:55:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:39274->10.242.244.80:8302: i/o timeout
2021/04/17 11:55:12 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:55:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:39424->10.242.244.80:8302: i/o timeout
2021/04/17 11:55:22 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:55:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:39594->10.242.244.80:8302: i/o timeout
2021/04/17 11:55:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:55:32 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:55:32 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:55:32 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:55:49 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:55:50 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:55:50 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:55:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:53158->10.242.244.80:8301: i/o timeout
2021/04/17 11:55:52 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:55:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:53168->10.242.244.80:8301: i/o timeout
2021/04/17 11:55:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:55:56 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:55:56 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:55:56 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:56:33 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 11:56:56 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:56:56 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:56:59 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 11:56:59 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:57:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:41068->10.242.244.80:8302: i/o timeout
2021/04/17 11:57:02 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:57:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:54316->10.242.244.80:8301: i/o timeout
2021/04/17 11:57:02 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:57:06 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 11:57:06 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 11:57:06 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 11:57:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:54388->10.242.244.80:8301: i/o timeout
2021/04/17 11:57:06 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 11:57:09 [ERR] memberlist: failed to receive: read tcp 10.242.146.73:8301->10.242.244.80:47244: i/o timeout from=10.242.244.80:47244
2021/04/17 11:57:09 [ERR] memberlist: Failed to send error: write tcp 10.242.146.73:8301->10.242.244.80:47244: i/o timeout from=10.242.244.80:47244
2021/04/17 11:57:11 [ERR] memberlist: failed to receive: read tcp 10.242.146.73:8301->10.242.244.80:47278: i/o timeout from=10.242.244.80:47278
2021/04/17 11:57:11 [ERR] memberlist: Failed to send error: write tcp 10.242.146.73:8301->10.242.244.80:47278: i/o timeout from=10.242.244.80:47278
2021/04/17 11:57:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:41236->10.242.244.80:8302: i/o timeout
2021/04/17 11:57:12 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:57:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:41486->10.242.244.80:8302: i/o timeout
2021/04/17 11:57:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:57:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:41646->10.242.244.80:8302: i/o timeout
2021/04/17 11:57:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:57:42 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 11:57:42 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:57:42 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:57:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:41798->10.242.244.80:8302: i/o timeout
2021/04/17 11:57:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:58:19 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:59:19 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 11:59:31 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 11:59:31 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 11:59:31 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 11:59:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:43692->10.242.244.80:8302: i/o timeout
2021/04/17 11:59:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 11:59:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:43930->10.242.244.80:8302: i/o timeout
2021/04/17 11:59:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:00:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:44080->10.242.244.80:8302: i/o timeout
2021/04/17 12:00:02 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:00:05 [ERR] memberlist: Push/Pull with dev-consul-server-2.dc1 failed: read tcp 10.242.146.73:43516->10.242.244.80:8302: i/o timeout
2021/04/17 12:00:12 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 12:00:12 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:00:12 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:00:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:44648->10.242.244.80:8302: i/o timeout
2021/04/17 12:00:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:00:43 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:00:54 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:00:54 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:00:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:59138->10.242.244.80:8301: i/o timeout
2021/04/17 12:00:56 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:00:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:59212->10.242.244.80:8301: i/o timeout
2021/04/17 12:00:59 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:01:00 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 12:01:00 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:01:00 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:01:01 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:01:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:59246->10.242.244.80:8301: i/o timeout
2021/04/17 12:01:02 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:01:53 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:02:11 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:02:11 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:02:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:48262->10.242.244.80:8302: i/o timeout
2021/04/17 12:02:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:02:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:48686->10.242.244.80:8302: i/o timeout
2021/04/17 12:02:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:02:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:48842->10.242.244.80:8302: i/o timeout
2021/04/17 12:02:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:02:47 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 12:02:47 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:02:47 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:03:01 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:03:14 [WARN] memberlist: Refuting a dead message (from: dev-consul-server-2)
2021/04/17 12:03:15 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:03:15 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:03:26 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:03:26 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:03:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:35550->10.242.244.80:8301: i/o timeout
2021/04/17 12:03:30 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:03:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:50572->10.242.244.80:8302: i/o timeout
2021/04/17 12:03:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:03:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:35614->10.242.244.80:8301: i/o timeout
2021/04/17 12:03:33 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:03:34 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 12:03:34 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:03:34 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:03:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:50736->10.242.244.80:8302: i/o timeout
2021/04/17 12:03:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:03:51 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 12:03:51 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:03:51 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:04:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:37056->10.242.244.80:8301: i/o timeout
2021/04/17 12:04:50 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:04:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:37096->10.242.244.80:8301: i/o timeout
2021/04/17 12:04:52 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:04:54 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 12:04:54 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:04:54 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:05:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:52362->10.242.244.80:8302: i/o timeout
2021/04/17 12:05:12 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:05:22 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/17 12:05:27 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 12:05:27 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:05:27 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:05:33 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:06:15 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:06:43 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:07:07 [ERR] memberlist: failed to receive: read tcp 10.242.146.73:8302->10.242.244.80:35630: i/o timeout from=10.242.244.80:35630
2021/04/17 12:07:07 [ERR] memberlist: Failed to send error: write tcp 10.242.146.73:8302->10.242.244.80:35630: i/o timeout from=10.242.244.80:35630
2021/04/17 12:07:39 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:07:39 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:07:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:39772->10.242.244.80:8301: i/o timeout
2021/04/17 12:07:42 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:07:45 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:07:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:39826->10.242.244.80:8301: i/o timeout
2021/04/17 12:07:46 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:07:46 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 12:07:46 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:07:46 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:08:11 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 12:08:45 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:08:50 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:09:45 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:09:53 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:10:39 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-zlv6m 10.242.253.199
2021/04/17 12:10:39 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:10:39 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:10:39 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-zlv6m 10.242.253.199
2021/04/17 12:10:45 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:10:45 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:10:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:42974->10.242.244.80:8301: i/o timeout
2021/04/17 12:10:47 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:10:50 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 12:10:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:43146->10.242.244.80:8301: i/o timeout
2021/04/17 12:10:53 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:10:53 [WARN] raft: Heartbeat timeout from "10.242.253.198:8300" reached, starting election
2021/04/17 12:10:53 [INFO] raft: Node at 10.242.146.73:8300 [Candidate] entering Candidate state in term 19
2021/04/17 12:10:53 [ERR] raft: Failed to make RequestVote RPC to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: EOF
2021/04/17 12:10:53 [INFO] raft: Node at 10.242.146.73:8300 [Follower] entering Follower state (Leader: "")
2021/04/17 12:10:53 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/17 12:10:54 [DEBUG] raft-net: 10.242.146.73:8300 accepted connection from: 10.242.253.198:33104
2021/04/17 12:10:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:43214->10.242.244.80:8301: i/o timeout
2021/04/17 12:10:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:10:56 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:10:56 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:11:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:58522->10.242.244.80:8302: i/o timeout
2021/04/17 12:11:07 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:11:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:58808->10.242.244.80:8302: i/o timeout
2021/04/17 12:11:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:11:27 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 12:11:27 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:11:27 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:11:45 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:12:03 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:12:55 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 12:12:55 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:12:55 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:12:55 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:12:55 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:12:56 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 12:12:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:46884->10.242.244.80:8301: i/o timeout
2021/04/17 12:12:57 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:13:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:46990->10.242.244.80:8301: i/o timeout
2021/04/17 12:13:01 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:13:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:47108->10.242.244.80:8301: i/o timeout
2021/04/17 12:13:05 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:13:05 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 12:13:05 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:13:05 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:13:08 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/17 12:13:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:34362->10.242.244.80:8302: i/o timeout
2021/04/17 12:13:22 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:13:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:34664->10.242.244.80:8302: i/o timeout
2021/04/17 12:13:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:13:35 [ERR] memberlist: Push/Pull with dev-consul-server-2.dc1 failed: read tcp 10.242.146.73:33918->10.242.244.80:8302: i/o timeout
2021/04/17 12:13:39 [ERR] memberlist: failed to receive: read tcp 10.242.146.73:8302->10.242.244.80:39740: i/o timeout from=10.242.244.80:39740
2021/04/17 12:13:39 [ERR] memberlist: Failed to send error: write tcp 10.242.146.73:8302->10.242.244.80:39740: i/o timeout from=10.242.244.80:39740
2021/04/17 12:13:47 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 12:13:47 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:13:47 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:13:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:35122->10.242.244.80:8302: i/o timeout
2021/04/17 12:13:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:14:13 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:14:15 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:14:41 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:14:41 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:14:41 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:14:41 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:14:43 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 12:14:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:50300->10.242.244.80:8301: i/o timeout
2021/04/17 12:14:48 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:14:51 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 12:14:51 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:14:51 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:14:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:50386->10.242.244.80:8301: i/o timeout
2021/04/17 12:14:51 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:14:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:37144->10.242.244.80:8302: i/o timeout
2021/04/17 12:14:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:14:52 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/17 12:14:52 [DEBUG] raft-net: 10.242.146.73:8300 accepted connection from: 10.242.253.198:37770
2021/04/17 12:15:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:37434->10.242.244.80:8302: i/o timeout
2021/04/17 12:15:02 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:15:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:37730->10.242.244.80:8302: i/o timeout
2021/04/17 12:15:12 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:15:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:37892->10.242.244.80:8302: i/o timeout
2021/04/17 12:15:22 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:15:22 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 12:15:22 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:15:22 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:15:58 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:15:58 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:15:58 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:15:58 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:15:58 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 12:15:58 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 12:16:07 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 12:16:07 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/17 12:16:08 [DEBUG] raft-net: 10.242.146.73:8300 accepted connection from: 10.242.253.198:39238
2021/04/17 12:16:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:52734->10.242.244.80:8301: i/o timeout
2021/04/17 12:16:19 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:16:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:52744->10.242.244.80:8301: i/o timeout
2021/04/17 12:16:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:16:22 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:16:22 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:16:23 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:16:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:39674->10.242.244.80:8302: i/o timeout
2021/04/17 12:16:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:16:43 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:16:43 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 12:16:43 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:16:43 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 12:16:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:53220->10.242.244.80:8301: i/o timeout
2021/04/17 12:16:51 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:16:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:53278->10.242.244.80:8301: i/o timeout
2021/04/17 12:16:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:16:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:53314->10.242.244.80:8301: i/o timeout
2021/04/17 12:16:58 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:16:59 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 12:16:59 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:16:59 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:17:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:53344->10.242.244.80:8301: i/o timeout
2021/04/17 12:17:01 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:17:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:40132->10.242.244.80:8302: i/o timeout
2021/04/17 12:17:02 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:17:06 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:17:06 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:17:06 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 12:17:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:53464->10.242.244.80:8301: i/o timeout
2021/04/17 12:17:09 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:17:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:53514->10.242.244.80:8301: i/o timeout
2021/04/17 12:17:11 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:17:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:40280->10.242.244.80:8302: i/o timeout
2021/04/17 12:17:12 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:17:13 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 12:17:13 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:17:13 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:17:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:53558->10.242.244.80:8301: i/o timeout
2021/04/17 12:17:14 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:17:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:40360->10.242.244.80:8302: i/o timeout
2021/04/17 12:17:22 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:17:36 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 12:17:37 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/17 12:17:42 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 12:17:42 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:17:42 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:17:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:40906->10.242.244.80:8302: i/o timeout
2021/04/17 12:17:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:18:12 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:18:33 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:19:13 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:19:42 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:19:53 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:20:02 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 12:20:03 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 12:20:04 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:20:04 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:20:05 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:20:05 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:20:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:56364->10.242.244.80:8301: i/o timeout
2021/04/17 12:20:10 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:20:10 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 12:20:10 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:20:10 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:20:16 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/17 12:20:16 [DEBUG] raft-net: 10.242.146.73:8300 accepted connection from: 10.242.253.198:44024
2021/04/17 12:20:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:43150->10.242.244.80:8302: i/o timeout
2021/04/17 12:20:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:20:20 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 12:20:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:43432->10.242.244.80:8302: i/o timeout
2021/04/17 12:20:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:20:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:43580->10.242.244.80:8302: i/o timeout
2021/04/17 12:20:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:20:47 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 12:20:47 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:20:47 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:21:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:22:03 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:23:22 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:23:22 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:23:35 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/17 12:23:36 [DEBUG] raft-net: 10.242.146.73:8300 accepted connection from: 10.242.253.198:47940
2021/04/17 12:23:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:47162->10.242.244.80:8302: i/o timeout
2021/04/17 12:23:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:23:43 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:23:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:47704->10.242.244.80:8302: i/o timeout
2021/04/17 12:23:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:23:58 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 12:24:01 [DEBUG] raft-net: 10.242.146.73:8300 accepted connection from: 10.242.244.80:55352
2021/04/17 12:24:01 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 12:24:02 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 12:24:02 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:24:02 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:24:02 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/17 12:24:05 [WARN] raft: Heartbeat timeout from "10.242.253.198:8300" reached, starting election
2021/04/17 12:24:05 [INFO] raft: Node at 10.242.146.73:8300 [Candidate] entering Candidate state in term 27
2021/04/17 12:24:05 [ERR] raft: Failed to make RequestVote RPC to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: EOF
2021/04/17 12:24:05 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:24:05 [INFO] raft: Election won. Tally: 2
2021/04/17 12:24:05 [INFO] raft: Node at 10.242.146.73:8300 [Leader] entering Leader state
2021/04/17 12:24:05 [INFO] raft: Added peer e3d1abd7-c4ef-d7bc-57e1-b271f273a71b, starting replication
2021/04/17 12:24:05 [INFO] raft: Added peer a06ee723-0a7a-bed8-b418-2a3a48a30906, starting replication
2021/04/17 12:24:05 [INFO] consul: cluster leadership acquired
2021/04/17 12:24:05 [INFO] consul: New leader elected: dev-consul-server-1
2021/04/17 12:24:05 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:24:05 [INFO] raft: pipelining replication to peer {Voter e3d1abd7-c4ef-d7bc-57e1-b271f273a71b 10.242.253.198:8300}
2021/04/17 12:24:06 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:24:08 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.502201272s
2021/04/17 12:24:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:24:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:10 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:24:10 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:24:10 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.984937684s
2021/04/17 12:24:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:13 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.449682433s
2021/04/17 12:24:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:15 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.146.73:49836->10.242.244.80:8300: i/o timeout
2021/04/17 12:24:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:15 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:24:16 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:49870->10.242.244.80:8300: i/o timeout
2021/04/17 12:24:17 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:24:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:48474->10.242.244.80:8302: i/o timeout
2021/04/17 12:24:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:24:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:25 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.146.73:50160->10.242.244.80:8300: i/o timeout
2021/04/17 12:24:25 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:24:27 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:50196->10.242.244.80:8300: i/o timeout
2021/04/17 12:24:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:48628->10.242.244.80:8302: i/o timeout
2021/04/17 12:24:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:24:27 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:24:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:35 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.253.198
2021/04/17 12:24:35 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.253.198:8300) (DC: dc1)
2021/04/17 12:24:35 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/04/17 12:24:35 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.253.198
2021/04/17 12:24:35 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.253.198:8300) (DC: dc1)
2021/04/17 12:24:35 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/04/17 12:24:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:35 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.146.73:50454->10.242.244.80:8300: i/o timeout
2021/04/17 12:24:35 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:24:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:49082->10.242.244.80:8302: i/o timeout
2021/04/17 12:24:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:24:37 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:50516->10.242.244.80:8300: i/o timeout
2021/04/17 12:24:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:38 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:24:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:45 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.146.73:50762->10.242.244.80:8300: i/o timeout
2021/04/17 12:24:45 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:24:47 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 12:24:47 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:24:47 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:24:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:47 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:24:47 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:24:48 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:50838->10.242.244.80:8300: i/o timeout
2021/04/17 12:24:49 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:24:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:24:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:49532->10.242.244.80:8302: i/o timeout
2021/04/17 12:24:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:24:53 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:24:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:55 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.146.73:51062->10.242.244.80:8300: i/o timeout
2021/04/17 12:24:55 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:24:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:59 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:51164->10.242.244.80:8300: i/o timeout
2021/04/17 12:24:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:24:59 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:25:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:05 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.146.73:51370->10.242.244.80:8300: i/o timeout
2021/04/17 12:25:06 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:25:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:09 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:51476->10.242.244.80:8300: i/o timeout
2021/04/17 12:25:10 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:25:11 [ERR] memberlist: failed to receive: read tcp 10.242.146.73:8302->10.242.244.80:46836: i/o timeout from=10.242.244.80:46836
2021/04/17 12:25:11 [ERR] memberlist: Failed to send error: write tcp 10.242.146.73:8302->10.242.244.80:46836: i/o timeout from=10.242.244.80:46836
2021/04/17 12:25:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:16 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.146.73:51668->10.242.244.80:8300: i/o timeout
2021/04/17 12:25:16 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:25:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:20 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:51788->10.242.244.80:8300: i/o timeout
2021/04/17 12:25:21 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:25:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:26 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.146.73:51966->10.242.244.80:8300: i/o timeout
2021/04/17 12:25:27 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:25:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:29 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:25:29 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:25:31 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:52116->10.242.244.80:8300: i/o timeout
2021/04/17 12:25:32 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:25:32 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 12:25:33 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:25:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:25:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:37 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.146.73:52274->10.242.244.80:8300: i/o timeout
2021/04/17 12:25:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:38 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:25:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:42 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.146.73:8300
2021/04/17 12:25:43 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:52460->10.242.244.80:8300: i/o timeout
2021/04/17 12:25:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:44 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:25:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:48 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.146.73:52608->10.242.244.80:8300: i/o timeout
2021/04/17 12:25:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:50 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:25:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:54 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:52816->10.242.244.80:8300: i/o timeout
2021/04/17 12:25:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:25:58 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:25:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:00 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.146.73:52996->10.242.244.80:8300: i/o timeout
2021/04/17 12:26:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:06 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:26:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:08 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:53202->10.242.244.80:8300: i/o timeout
2021/04/17 12:26:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:11 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:26:11 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:26:13 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:26:14 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:26:15 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:26:15 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:26:15 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 12:26:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:16 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.146.73:53438->10.242.244.80:8300: i/o timeout
2021/04/17 12:26:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:37182->10.242.244.80:8301: i/o timeout
2021/04/17 12:26:20 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:26:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:22 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 12:26:22 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:26:22 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:26:22 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 12:26:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:37274->10.242.244.80:8301: i/o timeout
2021/04/17 12:26:23 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:26:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:24 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:53678->10.242.244.80:8300: i/o timeout
2021/04/17 12:26:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:26 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:26:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:28 [ERR] memberlist: Push/Pull with dev-consul-server-2 failed: read tcp 10.242.146.73:37134->10.242.244.80:8301: i/o timeout
2021/04/17 12:26:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:33 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:26:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:35 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:26:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:36 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.146.73:54054->10.242.244.80:8300: i/o timeout
2021/04/17 12:26:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:45 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:54320->10.242.244.80:8300: i/o timeout
2021/04/17 12:26:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:46 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:26:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:53 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:26:53 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:26:55 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:26:55 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:56194
2021/04/17 12:26:56 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:26:56 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.146.73:54776->10.242.244.80:8300: i/o timeout
2021/04/17 12:26:56 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:26:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:26:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:27:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:06 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:55102->10.242.244.80:8300: i/o timeout
2021/04/17 12:27:06 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:27:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:13 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:27:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:16 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.146.73:55506->10.242.244.80:8300: i/o timeout
2021/04/17 12:27:17 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:27:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:27 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:27:27 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:55818->10.242.244.80:8300: i/o timeout
2021/04/17 12:27:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:35 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:27:35 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:27:37 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.146.73:56112->10.242.244.80:8300: i/o timeout
2021/04/17 12:27:38 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:27:38 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:27:38 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/17 12:27:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:47 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:27:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:48 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:56436->10.242.244.80:8300: i/o timeout
2021/04/17 12:27:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:57 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.146.73:56698->10.242.244.80:8300: i/o timeout
2021/04/17 12:27:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:27:59 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:27:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:28:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:07 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:28:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:09 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:57040->10.242.244.80:8300: i/o timeout
2021/04/17 12:28:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:17 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.146.73:57284->10.242.244.80:8300: i/o timeout
2021/04/17 12:28:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:17 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:28:17 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:28:20 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:28:20 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:28:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/17 12:28:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:27 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:28:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:30 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:57656->10.242.244.80:8300: i/o timeout
2021/04/17 12:28:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:37 [ERR] raft: Failed to AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: read tcp 10.242.146.73:57868->10.242.244.80:8300: i/o timeout
2021/04/17 12:28:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:41 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:28:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:47 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:28:49 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.146.73:8300
2021/04/17 12:28:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:28:51 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:58426->10.242.244.80:8300: i/o timeout
2021/04/17 12:28:51 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:28:51 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:28:51 [ERR] raft: peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300} has newer term, stopping replication
2021/04/17 12:28:51 [INFO] raft: Node at 10.242.146.73:8300 [Follower] entering Follower state (Leader: "")
2021/04/17 12:28:51 [INFO] raft: aborting pipeline replication to peer {Voter e3d1abd7-c4ef-d7bc-57e1-b271f273a71b 10.242.253.198:8300}
2021/04/17 12:28:51 [INFO] consul: cluster leadership lost
2021/04/17 12:28:51 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:28:51 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 12:28:51 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:28:59 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/17 12:28:59 [INFO] raft: Node at 10.242.146.73:8300 [Candidate] entering Candidate state in term 28
2021/04/17 12:28:59 [INFO] raft: Election won. Tally: 2
2021/04/17 12:28:59 [INFO] raft: Node at 10.242.146.73:8300 [Leader] entering Leader state
2021/04/17 12:28:59 [INFO] raft: Added peer e3d1abd7-c4ef-d7bc-57e1-b271f273a71b, starting replication
2021/04/17 12:28:59 [INFO] raft: Added peer a06ee723-0a7a-bed8-b418-2a3a48a30906, starting replication
2021/04/17 12:28:59 [INFO] consul: cluster leadership acquired
2021/04/17 12:28:59 [INFO] consul: New leader elected: dev-consul-server-1
2021/04/17 12:28:59 [INFO] raft: pipelining replication to peer {Voter e3d1abd7-c4ef-d7bc-57e1-b271f273a71b 10.242.253.198:8300}
2021/04/17 12:28:59 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 12:28:59 [ERR] raft: peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300} has newer term, stopping replication
2021/04/17 12:28:59 [INFO] raft: Node at 10.242.146.73:8300 [Follower] entering Follower state (Leader: "")
2021/04/17 12:28:59 [INFO] raft: aborting pipeline replication to peer {Voter e3d1abd7-c4ef-d7bc-57e1-b271f273a71b 10.242.253.198:8300}
2021/04/17 12:28:59 [INFO] consul: cluster leadership lost
2021/04/17 12:29:00 [ERR] agent: Coordinate update error: No cluster leader
2021/04/17 12:29:01 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since our last term is greater (28, 24)
2021/04/17 12:29:06 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.146.75:46008
2021/04/17 12:29:06 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/17 12:29:06 [INFO] raft: Node at 10.242.146.73:8300 [Candidate] entering Candidate state in term 32
2021/04/17 12:29:06 [INFO] raft: Election won. Tally: 2
2021/04/17 12:29:06 [INFO] raft: Node at 10.242.146.73:8300 [Leader] entering Leader state
2021/04/17 12:29:06 [INFO] raft: Added peer e3d1abd7-c4ef-d7bc-57e1-b271f273a71b, starting replication
2021/04/17 12:29:06 [INFO] raft: Added peer a06ee723-0a7a-bed8-b418-2a3a48a30906, starting replication
2021/04/17 12:29:06 [INFO] consul: New leader elected: dev-consul-server-1
2021/04/17 12:29:06 [INFO] consul: cluster leadership acquired
2021/04/17 12:29:06 [INFO] raft: pipelining replication to peer {Voter e3d1abd7-c4ef-d7bc-57e1-b271f273a71b 10.242.253.198:8300}
2021/04/17 12:29:07 [WARN] raft: AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300} rejected, sending older logs (next: 1306)
2021/04/17 12:29:08 [INFO] raft: pipelining replication to peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 12:31:13 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:31:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:45002->10.242.244.80:8301: i/o timeout
2021/04/17 12:31:14 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:31:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:31:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:31:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:45062->10.242.244.80:8301: i/o timeout
2021/04/17 12:31:18 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:31:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:31:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:45088->10.242.244.80:8301: i/o timeout
2021/04/17 12:31:21 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:31:21 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:31:22 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 12:31:22 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:31:22 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:31:22 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/17 12:31:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:31:23 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500555212s
2021/04/17 12:31:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:31:25 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.916867654s
2021/04/17 12:31:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:31:28 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 7.366146179s
2021/04/17 12:31:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:31:29 [ERR] raft: Failed to heartbeat to 10.242.244.80:8300: read tcp 10.242.146.73:58874->10.242.244.80:8300: i/o timeout
2021/04/17 12:31:30 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:31:30 [ERR] raft: Failed to pipeline AppendEntries to {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}: write tcp 10.242.146.73:58624->10.242.244.80:8300: use of closed network connection
2021/04/17 12:31:30 [INFO] raft: aborting pipeline replication to peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300}
2021/04/17 12:31:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:31:30 [WARN] Unable to get address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906, using fallback address 10.242.244.80:8300: Could not find address for server id a06ee723-0a7a-bed8-b418-2a3a48a30906
2021/04/17 12:31:31 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:31:31 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/17 12:31:33 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 2.500342589s
2021/04/17 12:31:33 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/17 12:31:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:31:35 [WARN] raft: Failed to contact a06ee723-0a7a-bed8-b418-2a3a48a30906 in 4.983119428s
2021/04/17 12:31:36 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:31:36 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:31:36 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/17 12:31:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/17 12:31:36 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.146.73:8300
2021/04/17 12:31:36 [ERR] raft: peer {Voter a06ee723-0a7a-bed8-b418-2a3a48a30906 10.242.244.80:8300} has newer term, stopping replication
2021/04/17 12:31:36 [INFO] raft: Node at 10.242.146.73:8300 [Follower] entering Follower state (Leader: "")
2021/04/17 12:31:36 [INFO] consul: cluster leadership lost
2021/04/17 12:31:36 [INFO] raft: aborting pipeline replication to peer {Voter e3d1abd7-c4ef-d7bc-57e1-b271f273a71b 10.242.253.198:8300}
2021/04/17 12:31:42 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/17 12:31:43 [DEBUG] raft-net: 10.242.146.73:8300 accepted connection from: 10.242.253.198:57246
2021/04/17 12:31:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:45548->10.242.244.80:8301: i/o timeout
2021/04/17 12:31:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:45602->10.242.244.80:8301: i/o timeout
2021/04/17 12:31:56 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:31:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:45638->10.242.244.80:8301: i/o timeout
2021/04/17 12:31:59 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:32:00 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 12:32:00 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:32:00 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:32:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:60670->10.242.244.80:8302: i/o timeout
2021/04/17 12:32:02 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:32:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:45668->10.242.244.80:8301: i/o timeout
2021/04/17 12:32:02 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:32:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:60816->10.242.244.80:8302: i/o timeout
2021/04/17 12:32:12 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:32:12 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:32:20 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:32:20 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:57408
2021/04/17 12:32:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:60964->10.242.244.80:8302: i/o timeout
2021/04/17 12:32:22 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:32:32 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 12:32:32 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:32:32 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:32:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:32888->10.242.244.80:8302: i/o timeout
2021/04/17 12:32:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:32:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:33:07 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 12:33:07 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:33:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:46844->10.242.244.80:8301: i/o timeout
2021/04/17 12:33:12 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/17 12:33:12 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/17 12:33:12 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.244.80
2021/04/17 12:33:12 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 12:33:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:35:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:35:53 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:36:29 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:37:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:38:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:38:33 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:38:33 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:38:33 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:38:43 [ERR] memberlist: failed to receive: read tcp 10.242.146.73:8301->10.242.244.80:44712: i/o timeout from=10.242.244.80:44712
2021/04/17 12:38:43 [ERR] memberlist: Failed to send error: write tcp 10.242.146.73:8301->10.242.244.80:44712: i/o timeout from=10.242.244.80:44712
2021/04/17 12:38:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:39694->10.242.244.80:8302: i/o timeout
2021/04/17 12:38:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:38:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:39840->10.242.244.80:8302: i/o timeout
2021/04/17 12:38:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:38:59 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/17 12:39:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.146.73:40406->10.242.244.80:8302: i/o timeout
2021/04/17 12:39:12 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/17 12:39:17 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/17 12:39:17 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 12:39:17 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 12:39:43 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:40:23 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:40:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:41:03 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:41:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:41:41 [ERR] memberlist: failed to receive: read tcp 10.242.146.73:8302->10.242.244.80:55662: i/o timeout from=10.242.244.80:55662
2021/04/17 12:41:41 [ERR] memberlist: Failed to send error: write tcp 10.242.146.73:8302->10.242.244.80:55662: i/o timeout from=10.242.244.80:55662
2021/04/17 12:41:41 [ERR] memberlist: failed to receive: read tcp 10.242.146.73:8302->10.242.244.80:55664: i/o timeout from=10.242.244.80:55664
2021/04/17 12:41:41 [ERR] memberlist: Failed to send error: write tcp 10.242.146.73:8302->10.242.244.80:55664: i/o timeout from=10.242.244.80:55664
2021/04/17 12:41:43 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:42:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:43:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:44:23 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:44:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:45:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:46:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:46:54 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:47:31 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/17 12:48:03 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:48:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:48:29 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:48:29 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.244.80:38456
2021/04/17 12:49:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:49:13 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:50:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:50:23 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:51:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:51:33 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:52:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:52:05 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 12:52:49 [ERR] memberlist: failed to receive: read tcp 10.242.146.73:8302->10.242.244.80:58400: i/o timeout from=10.242.244.80:58400
2021/04/17 12:52:49 [ERR] memberlist: Failed to send error: write tcp 10.242.146.73:8302->10.242.244.80:58400: i/o timeout from=10.242.244.80:58400
2021/04/17 12:53:13 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:53:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:55:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:56:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:56:53 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 12:57:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:58:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 12:59:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:00:03 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:00:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:00:31 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/17 13:01:13 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:01:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:01:53 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:02:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:03:33 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:04:13 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:05:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:05:23 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:06:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:07:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:08:03 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:08:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:08:43 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:09:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:11:53 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:12:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:14:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:14:33 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:16:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:17:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:17:43 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:19:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:20:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:20:53 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:21:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:21:33 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:22:43 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:23:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:25:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:26:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:27:23 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:27:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:28:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:29:03 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:29:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:31:43 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:32:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:34:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:34:53 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:35:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:36:03 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:36:43 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:37:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:37:23 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:38:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:39:03 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:39:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:40:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:41:43 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:42:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:42:23 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:43:03 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:43:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:44:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:46:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:46:13 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:47:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:49:23 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:50:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:50:33 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:51:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:52:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:52:43 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:54:23 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:54:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:55:03 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:55:43 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:56:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:56:23 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:57:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:57:33 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 13:58:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 13:59:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:00:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:01:13 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:02:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:02:53 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:03:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:04:03 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:04:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:04:43 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:05:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:06:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:06:53 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:08:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:08:33 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:09:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:10:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:10:43 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:11:23 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:12:03 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:12:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:13:13 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:13:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:14:53 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:15:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:16:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:18:03 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:18:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:18:43 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:19:53 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:20:33 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:21:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:21:13 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:22:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:23:53 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:24:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:25:33 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:25:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:27:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:28:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:30:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:30:43 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:31:23 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:32:33 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:32:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:34:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:34:43 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:35:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:36:53 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:37:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:38:03 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:38:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:39:13 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:40:04 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:40:23 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:41:33 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.244.80:8301
2021/04/17 14:42:34 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.244.80:8302
2021/04/17 14:43:12 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300
2021/04/17 14:43:13 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.244.80
2021/04/17 14:43:13 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/17 14:43:13 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.244.80
2021/04/17 14:43:13 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.244.80:8300) (DC: dc1)
2021/04/17 14:43:18 [WARN] raft: Rejecting vote request from 10.242.244.80:8300 since we have a leader: 10.242.253.198:8300