By type
+ CONSUL_BIND=
+ [ -n ]
+ CONSUL_CLIENT=
+ [ -n ]
+ CONSUL_DATA_DIR=/consul/data
+ CONSUL_CONFIG_DIR=/consul/config
+ [ -n ]
+ [ a = - ]
+ [ agent = agent ]
+ shift
+ set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
+ [ consul = consul ]
+ stat -c %u /consul/data
+ id -u consul
+ [ 100 != 100 ]
+ stat -c %u /consul/config
+ id -u consul
+ [ 100 != 100 ]
+ [ ! -z ]
+ exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
==> Starting Consul agent...
bootstrap_expect > 0: expecting 3 servers
==> Consul agent running!
Version: 'v1.0.6'
Node ID: '0a55ae1c-93ce-8ff5-8070-a7aec028a14a'
Node name: 'dev-consul-server-2'
Datacenter: 'dc1' (Segment: '')
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.242.37.206 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2021/05/02 10:40:43 [INFO] raft: Initial configuration (index=0): []
2021/05/02 10:40:43 [INFO] raft: Node at 10.242.37.206:8300 [Follower] entering Follower state (Leader: "")
2021/05/02 10:40:43 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.37.206
2021/05/02 10:40:43 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/05/02 10:40:43 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.37.206
2021/05/02 10:40:43 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/05/02 10:40:43 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.37.206:8300) (DC: dc1)
2021/05/02 10:40:43 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2021/05/02 10:40:43 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2021/05/02 10:40:43 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
2021/05/02 10:40:43 [INFO] agent: started state syncer
2021/05/02 10:40:43 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer
2021/05/02 10:40:43 [INFO] agent: Joining LAN cluster...
2021/05/02 10:40:43 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.198.2
2021/05/02 10:40:43 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-s67jk 10.242.77.196
2021/05/02 10:40:43 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 10:40:43 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.198.2:8300) (DC: dc1)
2021/05/02 10:40:43 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.242.37.206:8300,10.242.198.2:8300,10.242.134.13:8300
2021/05/02 10:40:43 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 10:40:43 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.134.13
2021/05/02 10:40:43 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.198.2
2021/05/02 10:40:43 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/02 10:40:43 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/02 10:40:43 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/05/02 10:40:43 [INFO] agent: (LAN) joined: 2 Err:
2021/05/02 10:40:43 [INFO] agent: Join LAN completed. Synced with 2 initial agents
2021/05/02 10:40:47 [DEBUG] raft-net: 10.242.37.206:8300 accepted connection from: 10.242.198.2:33834
2021/05/02 10:40:47 [DEBUG] raft-net: 10.242.37.206:8300 accepted connection from: 10.242.198.2:33836
2021/05/02 10:40:47 [INFO] agent: Synced node info
2021/05/02 10:40:48 [INFO] consul: New leader elected: dev-consul-server-0
==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=ae6b199e-ba19-88e5-299b-76fcb6839aa3&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2021/05/02 11:03:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:36824->10.242.134.13:8301: i/o timeout
2021/05/02 11:03:41 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:03:47 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/02 11:03:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:36958->10.242.134.13:8301: i/o timeout
2021/05/02 11:03:52 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:04:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:37064->10.242.134.13:8301: i/o timeout
2021/05/02 11:04:00 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:04:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:37078->10.242.134.13:8301: i/o timeout
2021/05/02 11:04:03 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:04:04 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:04:04 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:04:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:54462->10.242.134.13:8302: i/o timeout
2021/05/02 11:04:08 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:04:13 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.134.13:8301
2021/05/02 11:04:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:54646->10.242.134.13:8302: i/o timeout
2021/05/02 11:04:23 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:04:26 [ERR] memberlist: failed to receive: read tcp 10.242.37.206:8301->10.242.134.13:38248: i/o timeout from=10.242.134.13:38248
2021/05/02 11:04:26 [ERR] memberlist: Failed to send error: write tcp 10.242.37.206:8301->10.242.134.13:38248: i/o timeout from=10.242.134.13:38248
2021/05/02 11:04:32 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/02 11:04:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:54762->10.242.134.13:8302: i/o timeout
2021/05/02 11:04:33 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:04:35 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:04:35 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:04:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:37508->10.242.134.13:8301: i/o timeout
2021/05/02 11:04:36 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:04:38 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/02 11:04:38 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.134.13
2021/05/02 11:04:38 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/02 11:04:39 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.134.13
2021/05/02 11:04:39 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/02 11:04:40 [DEBUG] raft-net: 10.242.37.206:8300 accepted connection from: 10.242.134.13:36434
2021/05/02 11:04:40 [WARN] raft: Rejecting vote request from 10.242.134.13:8300 since we have a leader: 10.242.198.2:8300
2021/05/02 11:04:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:37608->10.242.134.13:8301: i/o timeout
2021/05/02 11:04:43 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:04:46 [INFO] consul: New leader elected: dev-consul-server-0
2021/05/02 11:04:47 [DEBUG] raft-net: 10.242.37.206:8300 accepted connection from: 10.242.198.2:50482
2021/05/02 11:05:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:38428->10.242.134.13:8301: i/o timeout
2021/05/02 11:05:51 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:05:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:38510->10.242.134.13:8301: i/o timeout
2021/05/02 11:05:57 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:05:59 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:05:59 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:05:59 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:06:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:38546->10.242.134.13:8301: i/o timeout
2021/05/02 11:06:00 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:06:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:55886->10.242.134.13:8302: i/o timeout
2021/05/02 11:06:03 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:06:06 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:06:06 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:06:13 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/02 11:06:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:38754->10.242.134.13:8301: i/o timeout
2021/05/02 11:06:16 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:06:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:38796->10.242.134.13:8301: i/o timeout
2021/05/02 11:06:19 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:06:20 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:06:20 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:06:20 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:06:23 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:06:23 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:06:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:38860->10.242.134.13:8301: i/o timeout
2021/05/02 11:06:24 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:06:27 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 11:06:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:39000->10.242.134.13:8301: i/o timeout
2021/05/02 11:06:36 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:06:39 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:06:39 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:06:41 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:06:41 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:06:41 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/02 11:06:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:39110->10.242.134.13:8301: i/o timeout
2021/05/02 11:06:44 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:06:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:39200->10.242.134.13:8301: i/o timeout
2021/05/02 11:06:52 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:06:54 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:06:54 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:06:58 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:06:58 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:06:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:39298->10.242.134.13:8301: i/o timeout
2021/05/02 11:06:59 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:07:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:39332->10.242.134.13:8301: i/o timeout
2021/05/02 11:07:02 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:07:03 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:07:03 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:07:03 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:07:04 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 11:07:04 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:07:04 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:07:04 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/02 11:07:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:39398->10.242.134.13:8301: i/o timeout
2021/05/02 11:07:08 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:07:10 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:07:10 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:07:10 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:07:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:39422->10.242.134.13:8301: i/o timeout
2021/05/02 11:07:11 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:07:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:56828->10.242.134.13:8302: i/o timeout
2021/05/02 11:07:18 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:07:26 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:07:26 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:07:26 [WARN] raft: Rejecting vote request from 10.242.134.13:8300 since we have a leader: 10.242.198.2:8300
2021/05/02 11:07:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:39694->10.242.134.13:8301: i/o timeout
2021/05/02 11:07:32 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:07:35 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:07:35 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:07:35 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:07:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:39746->10.242.134.13:8301: i/o timeout
2021/05/02 11:07:36 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:07:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:57132->10.242.134.13:8302: i/o timeout
2021/05/02 11:07:43 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:07:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:57248->10.242.134.13:8302: i/o timeout
2021/05/02 11:07:53 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:07:53 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.134.13:8301
2021/05/02 11:08:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:57376->10.242.134.13:8302: i/o timeout
2021/05/02 11:08:03 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:08:13 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/02 11:08:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:57442->10.242.134.13:8302: i/o timeout
2021/05/02 11:08:13 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:08:13 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.134.13
2021/05/02 11:08:13 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/02 11:08:18 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:08:18 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:08:19 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.134.13
2021/05/02 11:08:19 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/02 11:09:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:40796->10.242.134.13:8301: i/o timeout
2021/05/02 11:09:01 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:09:01 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 11:09:05 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/02 11:09:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:40876->10.242.134.13:8301: i/o timeout
2021/05/02 11:09:08 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:09:08 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:09:08 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:09:08 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:09:10 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:09:10 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:09:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:40920->10.242.134.13:8301: i/o timeout
2021/05/02 11:09:42 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/02 11:09:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:41344->10.242.134.13:8301: i/o timeout
2021/05/02 11:09:45 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:09:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:41432->10.242.134.13:8301: i/o timeout
2021/05/02 11:09:52 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:09:54 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 11:09:55 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:09:55 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:09:55 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:09:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:41470->10.242.134.13:8301: i/o timeout
2021/05/02 11:09:56 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:10:03 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.134.13:8301
2021/05/02 11:10:05 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:10:05 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:10:16 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 11:10:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:41782->10.242.134.13:8301: i/o timeout
2021/05/02 11:10:20 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:10:22 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 11:10:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:41926->10.242.134.13:8301: i/o timeout
2021/05/02 11:10:32 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:10:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:41964->10.242.134.13:8301: i/o timeout
2021/05/02 11:10:35 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:10:36 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:10:36 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:10:37 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 11:10:37 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:10:37 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:10:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:42122->10.242.134.13:8301: i/o timeout
2021/05/02 11:10:48 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:10:50 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:10:50 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:10:50 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:10:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:42144->10.242.134.13:8301: i/o timeout
2021/05/02 11:10:51 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:10:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:59486->10.242.134.13:8302: i/o timeout
2021/05/02 11:10:53 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:10:55 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:10:55 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:11:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:42498->10.242.134.13:8301: i/o timeout
2021/05/02 11:11:18 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:11:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:42528->10.242.134.13:8301: i/o timeout
2021/05/02 11:11:21 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:11:22 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:11:22 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:11:22 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:11:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:59980->10.242.134.13:8302: i/o timeout
2021/05/02 11:11:33 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:11:34 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:11:34 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:11:34 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 11:11:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:42982->10.242.134.13:8301: i/o timeout
2021/05/02 11:12:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:43022->10.242.134.13:8301: i/o timeout
2021/05/02 11:12:00 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:12:02 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:12:02 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:12:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:60354->10.242.134.13:8302: i/o timeout
2021/05/02 11:12:03 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:12:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:60544->10.242.134.13:8302: i/o timeout
2021/05/02 11:12:18 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:12:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:60668->10.242.134.13:8302: i/o timeout
2021/05/02 11:12:28 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:12:33 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/02 11:12:33 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.134.13
2021/05/02 11:12:33 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/02 11:12:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:60788->10.242.134.13:8302: i/o timeout
2021/05/02 11:12:38 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:12:43 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.134.13:8302
2021/05/02 11:12:50 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:12:50 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:12:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:43690->10.242.134.13:8301: i/o timeout
2021/05/02 11:12:54 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:12:55 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:12:55 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:12:55 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:13:02 [ERR] memberlist: failed to receive: read tcp 10.242.37.206:8301->10.242.134.13:44246: i/o timeout from=10.242.134.13:44246
2021/05/02 11:13:02 [ERR] memberlist: Failed to send error: write tcp 10.242.37.206:8301->10.242.134.13:44246: i/o timeout from=10.242.134.13:44246
2021/05/02 11:13:05 [ERR] memberlist: failed to receive: read tcp 10.242.37.206:8302->10.242.134.13:40010: i/o timeout from=10.242.134.13:40010
2021/05/02 11:13:05 [ERR] memberlist: Failed to send error: write tcp 10.242.37.206:8302->10.242.134.13:40010: i/o timeout from=10.242.134.13:40010
2021/05/02 11:13:05 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 11:13:37 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.134.13:8301
2021/05/02 11:13:40 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.134.13
2021/05/02 11:13:40 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/02 11:13:41 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:13:41 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:13:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:44338->10.242.134.13:8301: i/o timeout
2021/05/02 11:13:48 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:13:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:44380->10.242.134.13:8301: i/o timeout
2021/05/02 11:13:51 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:13:52 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:13:52 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:13:52 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:13:53 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:13:53 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:13:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:44450->10.242.134.13:8301: i/o timeout
2021/05/02 11:13:56 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:14:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:44512->10.242.134.13:8301: i/o timeout
2021/05/02 11:14:01 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:14:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:44552->10.242.134.13:8301: i/o timeout
2021/05/02 11:14:04 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:14:04 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:14:04 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:14:04 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:14:05 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:14:05 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:14:05 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 11:15:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:45434->10.242.134.13:8301: i/o timeout
2021/05/02 11:15:16 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:15:19 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:15:19 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:15:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:45454->10.242.134.13:8301: i/o timeout
2021/05/02 11:15:19 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:15:20 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:15:20 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:15:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:45542->10.242.134.13:8301: i/o timeout
2021/05/02 11:15:24 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:15:26 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:15:26 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:15:26 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:15:26 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:16:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:46150->10.242.134.13:8301: i/o timeout
2021/05/02 11:16:07 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:16:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:46294->10.242.134.13:8301: i/o timeout
2021/05/02 11:16:15 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:16:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:46518->10.242.134.13:8301: i/o timeout
2021/05/02 11:16:32 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:16:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:46640->10.242.134.13:8301: i/o timeout
2021/05/02 11:16:40 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:16:44 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:16:44 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:16:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:46700->10.242.134.13:8301: i/o timeout
2021/05/02 11:16:45 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:16:48 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:16:48 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:18:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:47702->10.242.134.13:8301: i/o timeout
2021/05/02 11:18:06 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:18:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:47746->10.242.134.13:8301: i/o timeout
2021/05/02 11:18:09 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:18:10 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:18:10 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:18:10 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:18:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:47768->10.242.134.13:8301: i/o timeout
2021/05/02 11:18:12 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:18:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:36984->10.242.134.13:8302: i/o timeout
2021/05/02 11:18:23 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:18:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:37102->10.242.134.13:8302: i/o timeout
2021/05/02 11:18:33 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:18:41 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.134.13:8301
2021/05/02 11:18:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:37234->10.242.134.13:8302: i/o timeout
2021/05/02 11:18:48 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:18:53 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/02 11:18:53 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.134.13
2021/05/02 11:18:53 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/02 11:18:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:37424->10.242.134.13:8302: i/o timeout
2021/05/02 11:18:58 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:19:00 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:19:00 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:19:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:48406->10.242.134.13:8301: i/o timeout
2021/05/02 11:19:02 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:19:04 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.134.13
2021/05/02 11:19:04 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/02 11:19:04 [WARN] raft: Rejecting vote request from 10.242.134.13:8300 since we have a leader: 10.242.198.2:8300
2021/05/02 11:19:10 [WARN] raft: Rejecting vote request from 10.242.134.13:8300 since we have a leader: 10.242.198.2:8300
2021/05/02 11:19:11 [WARN] raft: Heartbeat timeout from "10.242.198.2:8300" reached, starting election
2021/05/02 11:19:11 [INFO] raft: Node at 10.242.37.206:8300 [Candidate] entering Candidate state in term 5
2021/05/02 11:19:11 [INFO] raft: Node at 10.242.37.206:8300 [Follower] entering Follower state (Leader: "")
2021/05/02 11:19:11 [INFO] consul: New leader elected: dev-consul-server-0
2021/05/02 11:19:12 [DEBUG] raft-net: 10.242.37.206:8300 accepted connection from: 10.242.198.2:37610
2021/05/02 11:22:21 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/02 11:22:23 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:22:23 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:22:23 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:22:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:39992->10.242.134.13:8302: i/o timeout
2021/05/02 11:22:23 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:22:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:40110->10.242.134.13:8302: i/o timeout
2021/05/02 11:22:33 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:22:34 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/02 11:22:34 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.134.13:49082
2021/05/02 11:22:46 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/02 11:22:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:40306->10.242.134.13:8302: i/o timeout
2021/05/02 11:22:48 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:22:51 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.134.13:8301
2021/05/02 11:22:53 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/02 11:22:53 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.134.13
2021/05/02 11:22:53 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/02 11:22:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:40362->10.242.134.13:8302: i/o timeout
2021/05/02 11:22:58 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:23:43 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.134.13:8302
2021/05/02 11:24:31 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.134.13:8301
2021/05/02 11:25:11 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.134.13:8301
2021/05/02 11:25:13 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.134.13:8302
2021/05/02 11:25:51 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.134.13:8301
2021/05/02 11:26:10 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:26:10 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:26:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:53766->10.242.134.13:8301: i/o timeout
2021/05/02 11:26:14 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:26:17 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:26:17 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:26:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:53792->10.242.134.13:8301: i/o timeout
2021/05/02 11:26:17 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:26:20 [ERR] memberlist: failed to receive: read tcp 10.242.37.206:8301->10.242.134.13:51940: i/o timeout from=10.242.134.13:51940
2021/05/02 11:26:20 [ERR] memberlist: Failed to send error: write tcp 10.242.37.206:8301->10.242.134.13:51940: i/o timeout from=10.242.134.13:51940
2021/05/02 11:26:31 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.134.13:8301
2021/05/02 11:26:40 [ERR] memberlist: failed to receive: read tcp 10.242.37.206:8302->10.242.134.13:47726: i/o timeout from=10.242.134.13:47726
2021/05/02 11:26:40 [ERR] memberlist: Failed to send error: write tcp 10.242.37.206:8302->10.242.134.13:47726: i/o timeout from=10.242.134.13:47726
2021/05/02 11:27:41 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.134.13:8301
2021/05/02 11:28:13 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.134.13:8302
2021/05/02 11:29:13 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.134.13:8302
2021/05/02 11:29:51 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.134.13:8301
2021/05/02 11:29:59 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/02 11:30:13 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.134.13:8302
2021/05/02 11:31:01 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.134.13:8301
2021/05/02 11:31:13 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.134.13:8302
2021/05/02 11:31:20 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 11:31:29 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.134.13
2021/05/02 11:31:29 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/02 11:31:31 [WARN] raft: Rejecting vote request from 10.242.134.13:8300 since we have a leader: 10.242.198.2:8300
2021/05/02 11:31:32 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:31:32 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:32:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:58312->10.242.134.13:8301: i/o timeout
2021/05/02 11:32:07 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:32:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:58360->10.242.134.13:8301: i/o timeout
2021/05/02 11:32:09 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:32:11 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:32:11 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:32:11 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:32:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:58402->10.242.134.13:8301: i/o timeout
2021/05/02 11:32:12 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:32:18 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:32:18 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:35:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:60608->10.242.134.13:8301: i/o timeout
2021/05/02 11:35:01 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:35:01 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 11:35:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:60780->10.242.134.13:8301: i/o timeout
2021/05/02 11:35:15 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:35:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:60818->10.242.134.13:8301: i/o timeout
2021/05/02 11:35:18 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:35:51 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 11:36:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:33374->10.242.134.13:8301: i/o timeout
2021/05/02 11:36:20 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:38:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:34998->10.242.134.13:8301: i/o timeout
2021/05/02 11:38:24 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:38:26 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:38:26 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:38:30 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:38:30 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:38:30 [WARN] raft: Rejecting vote request from 10.242.134.13:8300 since we have a leader: 10.242.198.2:8300
2021/05/02 11:38:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:35104->10.242.134.13:8301: i/o timeout
2021/05/02 11:38:33 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:38:37 [WARN] raft: Heartbeat timeout from "10.242.198.2:8300" reached, starting election
2021/05/02 11:38:37 [INFO] raft: Node at 10.242.37.206:8300 [Candidate] entering Candidate state in term 8
2021/05/02 11:38:37 [INFO] raft: Election won. Tally: 2
2021/05/02 11:38:37 [INFO] raft: Node at 10.242.37.206:8300 [Leader] entering Leader state
2021/05/02 11:38:37 [INFO] raft: Added peer 1bb5da00-1e1b-db26-1715-28d37e255e42, starting replication
2021/05/02 11:38:37 [INFO] raft: Added peer 1f88c670-199c-19b9-77d9-326c154e09ac, starting replication
2021/05/02 11:38:37 [INFO] consul: cluster leadership acquired
2021/05/02 11:38:37 [INFO] consul: New leader elected: dev-consul-server-2
2021/05/02 11:38:37 [INFO] raft: pipelining replication to peer {Voter 1bb5da00-1e1b-db26-1715-28d37e255e42 10.242.198.2:8300}
2021/05/02 11:38:40 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:38:41 [WARN] raft: AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300} rejected, sending older logs (next: 763)
2021/05/02 11:38:41 [INFO] raft: pipelining replication to peer {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}
2021/05/02 11:38:44 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:38:44 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/02 11:38:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:38:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:35324->10.242.134.13:8301: i/o timeout
2021/05/02 11:39:06 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:39:10 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:39:12 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:39:12 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/02 11:39:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:39:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:35650->10.242.134.13:8301: i/o timeout
2021/05/02 11:39:15 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:39:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:35676->10.242.134.13:8301: i/o timeout
2021/05/02 11:39:18 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:39:18 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:39:18 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/02 11:39:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:39:30 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:39:50 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:39:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/02 11:39:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:39:52 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500284386s
2021/05/02 11:39:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:36132->10.242.134.13:8301: i/o timeout
2021/05/02 11:39:53 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:39:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:39:54 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 4.923373214s
2021/05/02 11:39:55 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:39:55 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:39:55 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:39:55 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/02 11:39:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:36158->10.242.134.13:8301: i/o timeout
2021/05/02 11:39:55 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:39:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:39:56 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:39:56 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:39:56 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/02 11:39:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:36196->10.242.134.13:8301: i/o timeout
2021/05/02 11:39:58 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:39:58 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:39:58 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500154099s
2021/05/02 11:39:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:40:01 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 4.983981087s
2021/05/02 11:40:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:36230->10.242.134.13:8301: i/o timeout
2021/05/02 11:40:01 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:40:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:40:01 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:40:01 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:40:01 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:40:01 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/02 11:40:03 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 7.45463759s
2021/05/02 11:40:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:40:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:40:06 [ERR] raft: Failed to pipeline AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: write tcp 10.242.37.206:32876->10.242.134.13:8300: use of closed network connection
2021/05/02 11:40:06 [INFO] raft: aborting pipeline replication to peer {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}
2021/05/02 11:40:06 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:40:07 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:46224->10.242.134.13:8300: i/o timeout
2021/05/02 11:40:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:40:07 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:40:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:53636->10.242.134.13:8302: i/o timeout
2021/05/02 11:40:08 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:40:08 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500170841s
2021/05/02 11:40:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:40:10 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 11:40:10 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:40:10 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:40:10 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/02 11:40:11 [INFO] raft: pipelining replication to peer {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}
2021/05/02 11:40:44 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:40:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:40:50 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:40:51 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500284717s
2021/05/02 11:40:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:36880->10.242.134.13:8301: i/o timeout
2021/05/02 11:40:51 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:40:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:40:53 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:40:53 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:40:53 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:40:53 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/02 11:40:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:40:53 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 4.987488774s
2021/05/02 11:40:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:40:56 [ERR] raft: peer {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300} has newer term, stopping replication
2021/05/02 11:40:56 [INFO] raft: aborting pipeline replication to peer {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}
2021/05/02 11:40:56 [INFO] raft: Node at 10.242.37.206:8300 [Follower] entering Follower state (Leader: "")
2021/05/02 11:40:56 [INFO] raft: aborting pipeline replication to peer {Voter 1bb5da00-1e1b-db26-1715-28d37e255e42 10.242.198.2:8300}
2021/05/02 11:40:56 [INFO] consul: cluster leadership lost
2021/05/02 11:40:56 [WARN] raft: Rejecting vote request from 10.242.134.13:8300 since our last index is greater (798, 797)
2021/05/02 11:40:56 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 11:40:56 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:40:56 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:41:02 [INFO] consul: New leader elected: dev-consul-server-0
2021/05/02 11:41:02 [DEBUG] raft-net: 10.242.37.206:8300 accepted connection from: 10.242.198.2:32880
2021/05/02 11:41:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:37056->10.242.134.13:8301: i/o timeout
2021/05/02 11:41:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:37256->10.242.134.13:8301: i/o timeout
2021/05/02 11:41:20 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:41:22 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:41:22 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:41:22 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:41:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:37286->10.242.134.13:8301: i/o timeout
2021/05/02 11:41:23 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:41:26 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:41:26 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:41:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:37380->10.242.134.13:8301: i/o timeout
2021/05/02 11:41:30 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:41:32 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:41:32 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:41:32 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:41:33 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:41:33 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:41:33 [WARN] raft: Rejecting vote request from 10.242.134.13:8300 since we have a leader: 10.242.198.2:8300
2021/05/02 11:41:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:37472->10.242.134.13:8301: i/o timeout
2021/05/02 11:41:37 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:41:39 [WARN] raft: Heartbeat timeout from "10.242.198.2:8300" reached, starting election
2021/05/02 11:41:39 [INFO] raft: Node at 10.242.37.206:8300 [Candidate] entering Candidate state in term 10
2021/05/02 11:41:39 [INFO] raft: Election won. Tally: 2
2021/05/02 11:41:39 [INFO] raft: Node at 10.242.37.206:8300 [Leader] entering Leader state
2021/05/02 11:41:39 [INFO] raft: Added peer 1bb5da00-1e1b-db26-1715-28d37e255e42, starting replication
2021/05/02 11:41:39 [INFO] raft: Added peer 1f88c670-199c-19b9-77d9-326c154e09ac, starting replication
2021/05/02 11:41:39 [INFO] consul: cluster leadership acquired
2021/05/02 11:41:39 [INFO] consul: New leader elected: dev-consul-server-2
2021/05/02 11:41:39 [INFO] raft: pipelining replication to peer {Voter 1bb5da00-1e1b-db26-1715-28d37e255e42 10.242.198.2:8300}
2021/05/02 11:41:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:37540->10.242.134.13:8301: i/o timeout
2021/05/02 11:41:41 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:41:42 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.501782949s
2021/05/02 11:41:42 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:41:42 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:41:42 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/02 11:41:42 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:41:43 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:41:43 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:41:43 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/02 11:41:43 [WARN] raft: AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300} rejected, sending older logs (next: 804)
2021/05/02 11:41:43 [INFO] raft: pipelining replication to peer {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}
2021/05/02 11:42:10 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:42:10 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/02 11:42:16 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:42:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:37988->10.242.134.13:8301: i/o timeout
2021/05/02 11:42:17 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:42:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:42:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:42:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:38064->10.242.134.13:8301: i/o timeout
2021/05/02 11:42:24 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:42:24 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/02 11:42:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:38096->10.242.134.13:8301: i/o timeout
2021/05/02 11:42:25 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:42:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:42:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:42:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:55438->10.242.134.13:8302: i/o timeout
2021/05/02 11:42:28 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:42:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:38134->10.242.134.13:8301: i/o timeout
2021/05/02 11:42:28 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:42:29 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:42:29 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:42:29 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:42:29 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/02 11:42:29 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 11:42:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:42:30 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:42:30 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:42:30 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:42:30 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/02 11:42:32 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:42:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:42:34 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500244766s
2021/05/02 11:42:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:38218->10.242.134.13:8301: i/o timeout
2021/05/02 11:42:34 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:42:36 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:42:40 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500210129s
2021/05/02 11:42:40 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:42:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:38314->10.242.134.13:8301: i/o timeout
2021/05/02 11:42:41 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:42:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:42:43 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 4.950428196s
2021/05/02 11:42:43 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:42:43 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:42:43 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:42:43 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/02 11:42:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:42:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:38356->10.242.134.13:8301: i/o timeout
2021/05/02 11:42:44 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:42:45 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 7.371823509s
2021/05/02 11:42:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:42:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:42:48 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/02 11:42:48 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:42:50 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500244896s
2021/05/02 11:42:50 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:42:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:42:53 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 4.968872068s
2021/05/02 11:42:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:42:55 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 7.443157841s
2021/05/02 11:42:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:42:56 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:42:56 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:42:56 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/02 11:42:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:42:58 [ERR] raft: Failed to pipeline AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: write tcp 10.242.37.206:35232->10.242.134.13:8300: use of closed network connection
2021/05/02 11:42:58 [INFO] raft: aborting pipeline replication to peer {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}
2021/05/02 11:42:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:55828->10.242.134.13:8302: i/o timeout
2021/05/02 11:42:58 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:42:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:38528->10.242.134.13:8301: i/o timeout
2021/05/02 11:42:58 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:42:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:43:00 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500165933s
2021/05/02 11:43:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:38576->10.242.134.13:8301: i/o timeout
2021/05/02 11:43:01 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:43:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:43:02 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:43:02 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:43:02 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:43:02 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/02 11:43:03 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 4.97543258s
2021/05/02 11:43:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:43:05 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 7.411379085s
2021/05/02 11:43:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:43:07 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:35240->10.242.134.13:8300: i/o timeout
2021/05/02 11:43:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:43:08 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:34044->10.242.134.13:8300: i/o timeout
2021/05/02 11:43:08 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:43:08 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:43:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:55966->10.242.134.13:8302: i/o timeout
2021/05/02 11:43:08 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:43:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:43:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:43:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:43:14 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.134.13
2021/05/02 11:43:14 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/02 11:43:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:43:16 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.134.13
2021/05/02 11:43:16 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/02 11:43:16 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:43:16 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:43:16 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/02 11:43:17 [INFO] raft: pipelining replication to peer {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}
2021/05/02 11:43:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:43:20 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/02 11:43:20 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/02 11:43:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:38838->10.242.134.13:8301: i/o timeout
2021/05/02 11:43:21 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:43:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:43:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:38882->10.242.134.13:8301: i/o timeout
2021/05/02 11:43:24 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:43:28 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:43:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:43:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:38976->10.242.134.13:8301: i/o timeout
2021/05/02 11:43:32 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:43:32 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:43:32 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/02 11:43:33 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500198308s
2021/05/02 11:43:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:43:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:39006->10.242.134.13:8301: i/o timeout
2021/05/02 11:43:35 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:43:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:43:36 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:43:36 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:43:36 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:43:36 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/02 11:43:36 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500229215s
2021/05/02 11:43:37 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:43:37 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:43:37 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/02 11:44:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:39418->10.242.134.13:8301: i/o timeout
2021/05/02 11:44:06 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:44:06 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:44:10 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:44:11 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500250237s
2021/05/02 11:44:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:44:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:39494->10.242.134.13:8301: i/o timeout
2021/05/02 11:44:12 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:44:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:44:13 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 4.984131712s
2021/05/02 11:44:14 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:44:14 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:44:14 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/02 11:44:15 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:44:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:39508->10.242.134.13:8301: i/o timeout
2021/05/02 11:44:15 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:44:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:44:16 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:44:16 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:44:16 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:44:16 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/02 11:44:18 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:44:24 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:44:24 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/02 11:44:24 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/02 11:44:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:44:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:39722->10.242.134.13:8301: i/o timeout
2021/05/02 11:44:30 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:44:30 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:44:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:44:33 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500196342s
2021/05/02 11:44:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:57056->10.242.134.13:8302: i/o timeout
2021/05/02 11:44:33 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:44:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:44:34 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:44:34 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:44:34 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:44:34 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/02 11:44:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:39788->10.242.134.13:8301: i/o timeout
2021/05/02 11:44:35 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:44:35 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 4.964138584s
2021/05/02 11:44:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:44:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:44:38 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 7.43937391s
2021/05/02 11:44:39 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:36382->10.242.134.13:8300: i/o timeout
2021/05/02 11:44:39 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:44:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:44:40 [ERR] raft: Failed to pipeline AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: write tcp 10.242.37.206:36380->10.242.134.13:8300: use of closed network connection
2021/05/02 11:44:40 [INFO] raft: aborting pipeline replication to peer {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}
2021/05/02 11:44:40 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:44:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:44:42 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.134.13:8301
2021/05/02 11:44:43 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500066312s
2021/05/02 11:44:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:57198->10.242.134.13:8302: i/o timeout
2021/05/02 11:44:43 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:44:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:44:45 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 5.000198976s
2021/05/02 11:44:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:44:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:44:48 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 7.478753646s
2021/05/02 11:44:49 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:37574->10.242.134.13:8300: i/o timeout
2021/05/02 11:44:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:44:50 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:44:50 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:37588->10.242.134.13:8300: i/o timeout
2021/05/02 11:44:50 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:44:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:44:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:57328->10.242.134.13:8302: i/o timeout
2021/05/02 11:44:53 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:44:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:44:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:44:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:44:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:00 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:37706->10.242.134.13:8300: i/o timeout
2021/05/02 11:45:00 [ERR] memberlist: Push/Pull with dev-consul-server-1.dc1 failed: read tcp 10.242.37.206:57046->10.242.134.13:8302: i/o timeout
2021/05/02 11:45:00 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:37714->10.242.134.13:8300: i/o timeout
2021/05/02 11:45:00 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:45:01 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:45:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:57464->10.242.134.13:8302: i/o timeout
2021/05/02 11:45:03 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:45:03 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/02 11:45:03 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.134.13
2021/05/02 11:45:03 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/02 11:45:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:08 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:45:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/02 11:45:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:10 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:37850->10.242.134.13:8300: i/o timeout
2021/05/02 11:45:10 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:45:11 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:37856->10.242.134.13:8300: i/o timeout
2021/05/02 11:45:11 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:45:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:20 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:37980->10.242.134.13:8300: i/o timeout
2021/05/02 11:45:20 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:45:21 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:37992->10.242.134.13:8300: i/o timeout
2021/05/02 11:45:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:22 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:45:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:29 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.134.13:8302
2021/05/02 11:45:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:30 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:38112->10.242.134.13:8300: i/o timeout
2021/05/02 11:45:31 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:45:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:32 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/02 11:45:32 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/02 11:45:32 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/02 11:45:32 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.134.13:56922
2021/05/02 11:45:32 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:38136->10.242.134.13:8300: i/o timeout
2021/05/02 11:45:33 [WARN] raft: Rejecting vote request from 10.242.134.13:8300 since we have a leader: 10.242.37.206:8300
2021/05/02 11:45:33 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:45:34 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:45:34 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:45:35 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:45:35 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:45:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:37 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.50025661s
2021/05/02 11:45:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:40 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 4.936368682s
2021/05/02 11:45:41 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:38240->10.242.134.13:8300: i/o timeout
2021/05/02 11:45:41 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:45:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:42 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 7.408301457s
2021/05/02 11:45:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:45 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:38274->10.242.134.13:8300: i/o timeout
2021/05/02 11:45:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:46 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:45:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:51 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:38376->10.242.134.13:8300: i/o timeout
2021/05/02 11:45:51 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:45:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:56 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:38444->10.242.134.13:8300: i/o timeout
2021/05/02 11:45:57 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:45:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:45:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:01 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:38502->10.242.134.13:8300: i/o timeout
2021/05/02 11:46:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:02 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:46:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:07 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:38586->10.242.134.13:8300: i/o timeout
2021/05/02 11:46:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:08 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:46:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:12 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:38652->10.242.134.13:8300: i/o timeout
2021/05/02 11:46:13 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:46:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:13 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/02 11:46:13 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/02 11:46:16 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:46:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:18 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:38726->10.242.134.13:8300: i/o timeout
2021/05/02 11:46:19 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:46:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:22 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.134.13:8301
2021/05/02 11:46:23 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:38800->10.242.134.13:8300: i/o timeout
2021/05/02 11:46:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:26 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:46:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:29 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:38878->10.242.134.13:8300: i/o timeout
2021/05/02 11:46:29 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.134.13:8302
2021/05/02 11:46:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:30 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:46:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:36 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:38966->10.242.134.13:8300: i/o timeout
2021/05/02 11:46:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:40 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:39014->10.242.134.13:8300: i/o timeout
2021/05/02 11:46:41 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:46:41 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:46:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:48 [ERR] memberlist: failed to receive: read tcp 10.242.37.206:8302->10.242.134.13:59484: i/o timeout from=10.242.134.13:59484
2021/05/02 11:46:48 [ERR] memberlist: Failed to send error: write tcp 10.242.37.206:8302->10.242.134.13:59484: i/o timeout from=10.242.134.13:59484
2021/05/02 11:46:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:51 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:39166->10.242.134.13:8300: i/o timeout
2021/05/02 11:46:51 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:39168->10.242.134.13:8300: i/o timeout
2021/05/02 11:46:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:52 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:46:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:46:55 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/02 11:46:55 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/02 11:46:58 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:46:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:01 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:47:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:02 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:39304->10.242.134.13:8300: i/o timeout
2021/05/02 11:47:03 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:47:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:04 [WARN] raft: Rejecting vote request from 10.242.134.13:8300 since we have a leader: 10.242.37.206:8300
2021/05/02 11:47:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:11 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:39428->10.242.134.13:8300: i/o timeout
2021/05/02 11:47:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:12 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/02 11:47:12 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:47:12 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:47:12 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/02 11:47:13 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.134.13
2021/05/02 11:47:13 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/02 11:47:13 [WARN] raft: Rejecting vote request from 10.242.134.13:8300 since we have a leader: 10.242.37.206:8300
2021/05/02 11:47:16 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:47:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:18 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/02 11:47:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:41990->10.242.134.13:8301: i/o timeout
2021/05/02 11:47:22 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:47:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:42018->10.242.134.13:8301: i/o timeout
2021/05/02 11:47:24 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:47:24 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:47:25 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500383324s
2021/05/02 11:47:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:42048->10.242.134.13:8301: i/o timeout
2021/05/02 11:47:27 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:47:27 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 4.964263152s
2021/05/02 11:47:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:28 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:47:28 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:47:28 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/02 11:47:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:30 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 7.445885538s
2021/05/02 11:47:31 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:39696->10.242.134.13:8300: i/o timeout
2021/05/02 11:47:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:32 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.134.13:8301
2021/05/02 11:47:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:59418->10.242.134.13:8302: i/o timeout
2021/05/02 11:47:33 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:47:33 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:39456->10.242.134.13:8300: i/o timeout
2021/05/02 11:47:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:34 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:47:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:37 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/02 11:47:37 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/02 11:47:40 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:47:40 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/02 11:47:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:42 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:47:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:59556->10.242.134.13:8302: i/o timeout
2021/05/02 11:47:43 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:47:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:44 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:39858->10.242.134.13:8300: i/o timeout
2021/05/02 11:47:45 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:47:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:52 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:39964->10.242.134.13:8300: i/o timeout
2021/05/02 11:47:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:59624->10.242.134.13:8302: i/o timeout
2021/05/02 11:47:53 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:47:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:55 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:40004->10.242.134.13:8300: i/o timeout
2021/05/02 11:47:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:47:56 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:47:57 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:47:57 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:47:57 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/02 11:47:57 [WARN] raft: Rejecting vote request from 10.242.134.13:8300 since we have a leader: 10.242.37.206:8300
2021/05/02 11:48:02 [ERR] raft: peer {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300} has newer term, stopping replication
2021/05/02 11:48:02 [INFO] raft: Node at 10.242.37.206:8300 [Follower] entering Follower state (Leader: "")
2021/05/02 11:48:02 [WARN] consul.coordinate: Batch update failed: leadership lost while committing log
2021/05/02 11:48:02 [INFO] raft: aborting pipeline replication to peer {Voter 1bb5da00-1e1b-db26-1715-28d37e255e42 10.242.198.2:8300}
2021/05/02 11:48:02 [INFO] consul: cluster leadership lost
2021/05/02 11:48:04 [WARN] raft: Rejecting vote request from 10.242.134.13:8300 since our last index is greater (889, 848)
2021/05/02 11:48:08 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/05/02 11:48:08 [INFO] raft: Node at 10.242.37.206:8300 [Candidate] entering Candidate state in term 16
2021/05/02 11:48:08 [INFO] raft: Election won. Tally: 2
2021/05/02 11:48:08 [INFO] raft: Node at 10.242.37.206:8300 [Leader] entering Leader state
2021/05/02 11:48:08 [INFO] raft: Added peer 1bb5da00-1e1b-db26-1715-28d37e255e42, starting replication
2021/05/02 11:48:08 [INFO] raft: Added peer 1f88c670-199c-19b9-77d9-326c154e09ac, starting replication
2021/05/02 11:48:08 [INFO] consul: cluster leadership acquired
2021/05/02 11:48:08 [INFO] consul: New leader elected: dev-consul-server-2
2021/05/02 11:48:08 [WARN] raft: AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300} rejected, sending older logs (next: 849)
2021/05/02 11:48:08 [INFO] raft: pipelining replication to peer {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}
2021/05/02 11:48:08 [INFO] raft: pipelining replication to peer {Voter 1bb5da00-1e1b-db26-1715-28d37e255e42 10.242.198.2:8300}
2021/05/02 11:49:03 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:49:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:43338->10.242.134.13:8301: i/o timeout
2021/05/02 11:49:07 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:49:07 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500314207s
2021/05/02 11:49:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:43348->10.242.134.13:8301: i/o timeout
2021/05/02 11:49:09 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:49:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:10 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 4.975871442s
2021/05/02 11:49:11 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:49:11 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:49:11 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:49:11 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/02 11:49:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:12 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 7.459881845s
2021/05/02 11:49:13 [INFO] raft: aborting pipeline replication to peer {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}
2021/05/02 11:49:13 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:49:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:60710->10.242.134.13:8302: i/o timeout
2021/05/02 11:49:13 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:49:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:15 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500202323s
2021/05/02 11:49:16 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:40314->10.242.134.13:8300: i/o timeout
2021/05/02 11:49:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:16 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:49:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:18 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 4.954221154s
2021/05/02 11:49:18 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/02 11:49:18 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.134.13:34522
2021/05/02 11:49:19 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/02 11:49:19 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/02 11:49:20 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 7.412090382s
2021/05/02 11:49:21 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:49:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:23 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:41130->10.242.134.13:8300: i/o timeout
2021/05/02 11:49:23 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:49:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:26 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:41178->10.242.134.13:8300: i/o timeout
2021/05/02 11:49:27 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:49:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:60914->10.242.134.13:8302: i/o timeout
2021/05/02 11:49:28 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:49:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:33 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:41268->10.242.134.13:8300: i/o timeout
2021/05/02 11:49:33 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:49:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:37 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:41320->10.242.134.13:8300: i/o timeout
2021/05/02 11:49:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:38 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:49:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:42 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.134.13:8301
2021/05/02 11:49:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:43 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:41394->10.242.134.13:8300: i/o timeout
2021/05/02 11:49:43 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:49:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:32810->10.242.134.13:8302: i/o timeout
2021/05/02 11:49:43 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:49:43 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/02 11:49:43 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.134.13
2021/05/02 11:49:43 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/02 11:49:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:48 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:41458->10.242.134.13:8300: i/o timeout
2021/05/02 11:49:49 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:49:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:53 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:41532->10.242.134.13:8300: i/o timeout
2021/05/02 11:49:53 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:49:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:49:56 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.134.13
2021/05/02 11:49:56 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/02 11:49:56 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:49:56 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:49:56 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/02 11:49:56 [WARN] raft: Rejecting vote request from 10.242.134.13:8300 since we have a leader: 10.242.37.206:8300
2021/05/02 11:49:56 [ERR] raft: peer {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300} has newer term, stopping replication
2021/05/02 11:49:56 [INFO] raft: Node at 10.242.37.206:8300 [Follower] entering Follower state (Leader: "")
2021/05/02 11:49:56 [INFO] raft: aborting pipeline replication to peer {Voter 1bb5da00-1e1b-db26-1715-28d37e255e42 10.242.198.2:8300}
2021/05/02 11:49:56 [ERR] consul: failed to reconcile member: {dev-consul-server-1 10.242.134.13 8301 map[vsn_max:3 build:1.0.6:9a494b5f port:8300 role:consul dc:dc1 segment: wan_join_port:8302 vsn_min:2 raft_vsn:3 id:1f88c670-199c-19b9-77d9-326c154e09ac vsn:2 expect:3] alive 1 5 2 2 5 4}: leadership lost while committing log
2021/05/02 11:49:56 [INFO] consul: cluster leadership lost
2021/05/02 11:49:58 [WARN] consul.coordinate: Batch update failed: node is not the leader
2021/05/02 11:50:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:44052->10.242.134.13:8301: i/o timeout
2021/05/02 11:50:01 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:50:01 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/05/02 11:50:01 [INFO] raft: Node at 10.242.37.206:8300 [Candidate] entering Candidate state in term 17
2021/05/02 11:50:01 [INFO] raft: Election won. Tally: 2
2021/05/02 11:50:01 [INFO] raft: Node at 10.242.37.206:8300 [Leader] entering Leader state
2021/05/02 11:50:01 [INFO] consul: cluster leadership acquired
2021/05/02 11:50:01 [INFO] raft: Added peer 1bb5da00-1e1b-db26-1715-28d37e255e42, starting replication
2021/05/02 11:50:01 [INFO] raft: Added peer 1f88c670-199c-19b9-77d9-326c154e09ac, starting replication
2021/05/02 11:50:01 [INFO] consul: New leader elected: dev-consul-server-2
2021/05/02 11:50:01 [INFO] raft: pipelining replication to peer {Voter 1bb5da00-1e1b-db26-1715-28d37e255e42 10.242.198.2:8300}
2021/05/02 11:50:03 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/02 11:50:04 [WARN] raft: AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300} rejected, sending older logs (next: 903)
2021/05/02 11:50:04 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:50:04 [INFO] raft: pipelining replication to peer {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}
2021/05/02 11:53:10 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:53:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:46482->10.242.134.13:8301: i/o timeout
2021/05/02 11:53:13 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:53:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:46574->10.242.134.13:8301: i/o timeout
2021/05/02 11:53:20 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:53:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:53:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:53:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:46606->10.242.134.13:8301: i/o timeout
2021/05/02 11:53:23 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:53:24 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:53:24 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:53:24 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:53:24 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/02 11:53:24 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:53:25 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:53:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:53:25 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:53:27 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:53:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:53:28 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500223955s
2021/05/02 11:53:29 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:53:29 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:53:29 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/02 11:53:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:53:34 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:53:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:46760->10.242.134.13:8301: i/o timeout
2021/05/02 11:53:35 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:53:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:53:37 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500325208s
2021/05/02 11:53:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:53:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:35868->10.242.134.13:8302: i/o timeout
2021/05/02 11:53:38 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:53:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:46788->10.242.134.13:8301: i/o timeout
2021/05/02 11:53:38 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:53:39 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:53:39 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:53:39 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:53:39 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/02 11:53:39 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 4.99742875s
2021/05/02 11:53:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:53:41 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:53:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:53:43 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500205834s
2021/05/02 11:53:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:53:44 [ERR] raft: Failed to pipeline AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: write tcp 10.242.37.206:41604->10.242.134.13:8300: use of closed network connection
2021/05/02 11:53:44 [INFO] raft: aborting pipeline replication to peer {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}
2021/05/02 11:53:44 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:53:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:53:47 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:53:47 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/02 11:53:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:53:48 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500156522s
2021/05/02 11:53:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:53:50 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:53:50 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:53:50 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/02 11:53:51 [INFO] raft: pipelining replication to peer {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}
2021/05/02 11:53:52 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:53:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:53:54 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500196687s
2021/05/02 11:53:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:47058->10.242.134.13:8301: i/o timeout
2021/05/02 11:53:58 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:53:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:00 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500280708s
2021/05/02 11:54:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:47122->10.242.134.13:8301: i/o timeout
2021/05/02 11:54:02 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:54:03 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 4.925924849s
2021/05/02 11:54:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:36192->10.242.134.13:8302: i/o timeout
2021/05/02 11:54:03 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:54:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:04 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:54:04 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:54:04 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/02 11:54:05 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 7.409070451s
2021/05/02 11:54:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:08 [INFO] raft: aborting pipeline replication to peer {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}
2021/05/02 11:54:08 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:54:09 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:41662->10.242.134.13:8300: i/o timeout
2021/05/02 11:54:09 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/02 11:54:09 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.134.13:35728
2021/05/02 11:54:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:09 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:54:10 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500206105s
2021/05/02 11:54:10 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/02 11:54:10 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/02 11:54:12 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:54:13 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 4.998340806s
2021/05/02 11:54:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:36324->10.242.134.13:8302: i/o timeout
2021/05/02 11:54:13 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:54:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:15 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 7.475830732s
2021/05/02 11:54:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:18 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:44912->10.242.134.13:8300: i/o timeout
2021/05/02 11:54:18 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:54:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:19 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:44934->10.242.134.13:8300: i/o timeout
2021/05/02 11:54:20 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:54:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:22 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 11:54:22 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/02 11:54:22 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:54:22 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:54:22 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/02 11:54:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:24 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.50046168s
2021/05/02 11:54:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:47406->10.242.134.13:8301: i/o timeout
2021/05/02 11:54:24 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:54:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:26 [INFO] raft: pipelining replication to peer {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}
2021/05/02 11:54:27 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:54:27 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:54:27 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/02 11:54:27 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:54:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:47450->10.242.134.13:8301: i/o timeout
2021/05/02 11:54:27 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:54:28 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:54:28 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:54:28 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:54:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:29 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:54:30 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:54:30 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:54:30 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/02 11:54:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:47514->10.242.134.13:8301: i/o timeout
2021/05/02 11:54:32 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:54:32 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:54:32 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/02 11:54:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:47544->10.242.134.13:8301: i/o timeout
2021/05/02 11:54:35 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:54:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:35 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500302095s
2021/05/02 11:54:36 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:54:36 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:54:36 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:54:36 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/02 11:54:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:38 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 4.918683457s
2021/05/02 11:54:39 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:54:39 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:54:39 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/02 11:54:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:42 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500340845s
2021/05/02 11:54:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:47652->10.242.134.13:8301: i/o timeout
2021/05/02 11:54:42 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:54:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:36724->10.242.134.13:8302: i/o timeout
2021/05/02 11:54:43 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:54:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:47682->10.242.134.13:8301: i/o timeout
2021/05/02 11:54:45 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:54:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:45 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500216655s
2021/05/02 11:54:46 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:54:46 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:54:46 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:54:46 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/02 11:54:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:48 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 4.986394251s
2021/05/02 11:54:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:50 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 7.48111702s
2021/05/02 11:54:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:51 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/02 11:54:51 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/02 11:54:52 [INFO] raft: aborting pipeline replication to peer {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}
2021/05/02 11:54:52 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:54:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:36852->10.242.134.13:8302: i/o timeout
2021/05/02 11:54:53 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:54:54 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:45078->10.242.134.13:8300: i/o timeout
2021/05/02 11:54:54 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:54:55 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:54:55 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500264436s
2021/05/02 11:54:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:54:57 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 4.988072637s
2021/05/02 11:54:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:00 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 7.449681458s
2021/05/02 11:55:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:02 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:45500->10.242.134.13:8300: i/o timeout
2021/05/02 11:55:02 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:55:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:36992->10.242.134.13:8302: i/o timeout
2021/05/02 11:55:03 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:55:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:05 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:45538->10.242.134.13:8300: i/o timeout
2021/05/02 11:55:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:06 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:55:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:12 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:45640->10.242.134.13:8300: i/o timeout
2021/05/02 11:55:13 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:55:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:37068->10.242.134.13:8302: i/o timeout
2021/05/02 11:55:13 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:55:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:16 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:45690->10.242.134.13:8300: i/o timeout
2021/05/02 11:55:16 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:55:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:18 [ERR] memberlist: failed to receive: read tcp 10.242.37.206:8301->10.242.134.13:41228: i/o timeout from=10.242.134.13:41228
2021/05/02 11:55:18 [ERR] memberlist: Failed to send error: write tcp 10.242.37.206:8301->10.242.134.13:41228: i/o timeout from=10.242.134.13:41228
2021/05/02 11:55:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:23 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:45768->10.242.134.13:8300: i/o timeout
2021/05/02 11:55:23 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:55:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:26 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:45822->10.242.134.13:8300: i/o timeout
2021/05/02 11:55:27 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:55:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:37324->10.242.134.13:8302: i/o timeout
2021/05/02 11:55:28 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:55:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:33 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:45910->10.242.134.13:8300: i/o timeout
2021/05/02 11:55:33 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:55:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:33 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/02 11:55:33 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/02 11:55:36 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:55:37 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:45962->10.242.134.13:8300: i/o timeout
2021/05/02 11:55:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:38 [ERR] memberlist: failed to receive: read tcp 10.242.37.206:8302->10.242.134.13:37016: i/o timeout from=10.242.134.13:37016
2021/05/02 11:55:38 [ERR] memberlist: Failed to send error: write tcp 10.242.37.206:8302->10.242.134.13:37016: i/o timeout from=10.242.134.13:37016
2021/05/02 11:55:38 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:55:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:37454->10.242.134.13:8302: i/o timeout
2021/05/02 11:55:38 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:55:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:43 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:46032->10.242.134.13:8300: i/o timeout
2021/05/02 11:55:43 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:55:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:48 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:46104->10.242.134.13:8300: i/o timeout
2021/05/02 11:55:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:37584->10.242.134.13:8302: i/o timeout
2021/05/02 11:55:48 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:55:49 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:55:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:55:51 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/02 11:55:51 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:55:51 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:55:51 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/02 11:55:51 [INFO] raft: pipelining replication to peer {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}
2021/05/02 11:56:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:49050->10.242.134.13:8301: i/o timeout
2021/05/02 11:56:30 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:56:30 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:56:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:56:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:49092->10.242.134.13:8301: i/o timeout
2021/05/02 11:56:33 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:56:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:56:33 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500338244s
2021/05/02 11:56:34 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:56:34 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:56:34 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:56:34 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/02 11:56:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:56:36 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 4.955706488s
2021/05/02 11:56:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:56:37 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:56:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:38226->10.242.134.13:8302: i/o timeout
2021/05/02 11:56:38 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:56:39 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500246831s
2021/05/02 11:56:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:56:41 [INFO] raft: aborting pipeline replication to peer {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}
2021/05/02 11:56:41 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:56:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:56:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:56:43 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.500260369s
2021/05/02 11:56:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:56:45 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/02 11:56:45 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/02 11:56:46 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 4.912343187s
2021/05/02 11:56:47 [ERR] raft: Failed to heartbeat to 10.242.134.13:8300: read tcp 10.242.37.206:46252->10.242.134.13:8300: i/o timeout
2021/05/02 11:56:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:38362->10.242.134.13:8302: i/o timeout
2021/05/02 11:56:48 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:56:48 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 7.366780159s
2021/05/02 11:56:48 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/02 11:56:48 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:56:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:56:51 [ERR] raft: Failed to AppendEntries to {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300}: read tcp 10.242.37.206:46922->10.242.134.13:8300: i/o timeout
2021/05/02 11:56:51 [WARN] Unable to get address for server id 1f88c670-199c-19b9-77d9-326c154e09ac, using fallback address 10.242.134.13:8300: Could not find address for server id 1f88c670-199c-19b9-77d9-326c154e09ac
2021/05/02 11:56:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:56:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:56:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:56:56 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:56:56 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:56:56 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/02 11:56:57 [WARN] raft: Rejecting vote request from 10.242.134.13:8300 since we have a leader: 10.242.37.206:8300
2021/05/02 11:56:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/02 11:56:58 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/02 11:56:58 [WARN] raft: Failed to contact 1f88c670-199c-19b9-77d9-326c154e09ac in 2.50020358s
2021/05/02 11:56:58 [ERR] raft: peer {Voter 1f88c670-199c-19b9-77d9-326c154e09ac 10.242.134.13:8300} has newer term, stopping replication
2021/05/02 11:56:58 [INFO] raft: Node at 10.242.37.206:8300 [Follower] entering Follower state (Leader: "")
2021/05/02 11:56:58 [INFO] raft: aborting pipeline replication to peer {Voter 1bb5da00-1e1b-db26-1715-28d37e255e42 10.242.198.2:8300}
2021/05/02 11:56:58 [INFO] consul: cluster leadership lost
2021/05/02 11:56:59 [WARN] consul.coordinate: Batch update failed: node is not the leader
2021/05/02 11:57:00 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/02 11:57:05 [INFO] consul: New leader elected: dev-consul-server-0
2021/05/02 11:57:06 [DEBUG] raft-net: 10.242.37.206:8300 accepted connection from: 10.242.198.2:49190
2021/05/02 11:57:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:49562->10.242.134.13:8301: i/o timeout
2021/05/02 11:57:08 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:57:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:49616->10.242.134.13:8301: i/o timeout
2021/05/02 11:57:12 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:57:12 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:57:12 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:57:12 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:57:14 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:57:14 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:57:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:49654->10.242.134.13:8301: i/o timeout
2021/05/02 11:57:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:49692->10.242.134.13:8301: i/o timeout
2021/05/02 11:57:18 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:57:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:49726->10.242.134.13:8301: i/o timeout
2021/05/02 11:57:21 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:57:22 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 11:57:22 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:57:22 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:57:26 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:57:26 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:57:35 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 11:57:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:49926->10.242.134.13:8301: i/o timeout
2021/05/02 11:57:37 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:57:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:50006->10.242.134.13:8301: i/o timeout
2021/05/02 11:57:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:50040->10.242.134.13:8301: i/o timeout
2021/05/02 11:57:45 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:57:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:50066->10.242.134.13:8301: i/o timeout
2021/05/02 11:57:48 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 11:57:49 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 11:57:49 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:57:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:39274->10.242.134.13:8302: i/o timeout
2021/05/02 11:57:58 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:58:06 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/02 11:58:06 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.134.13:39676
2021/05/02 11:58:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:39412->10.242.134.13:8302: i/o timeout
2021/05/02 11:58:08 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:58:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:39466->10.242.134.13:8302: i/o timeout
2021/05/02 11:58:18 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:58:22 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.134.13:8301
2021/05/02 11:58:28 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/02 11:58:28 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.134.13
2021/05/02 11:58:28 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/02 11:58:29 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.134.13:8302
2021/05/02 11:58:32 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/02 11:58:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:39722->10.242.134.13:8302: i/o timeout
2021/05/02 11:58:33 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 11:58:39 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 11:58:39 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 11:58:39 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.134.13
2021/05/02 11:58:39 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/02 11:58:39 [WARN] raft: Rejecting vote request from 10.242.134.13:8300 since we have a leader: 10.242.198.2:8300
2021/05/02 12:04:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:54984->10.242.134.13:8301: i/o timeout
2021/05/02 12:04:13 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:04:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:55090->10.242.134.13:8301: i/o timeout
2021/05/02 12:04:21 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:04:25 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 12:04:25 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 12:04:25 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 12:04:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:55154->10.242.134.13:8301: i/o timeout
2021/05/02 12:04:26 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:04:27 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 12:04:27 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 12:04:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:55276->10.242.134.13:8301: i/o timeout
2021/05/02 12:04:36 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:04:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:55330->10.242.134.13:8301: i/o timeout
2021/05/02 12:04:39 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:04:43 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 12:04:43 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 12:04:43 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 12:04:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:55390->10.242.134.13:8301: i/o timeout
2021/05/02 12:04:44 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:04:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:44562->10.242.134.13:8302: i/o timeout
2021/05/02 12:04:53 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 12:05:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:44696->10.242.134.13:8302: i/o timeout
2021/05/02 12:05:03 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 12:05:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:44766->10.242.134.13:8302: i/o timeout
2021/05/02 12:05:13 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 12:05:13 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/02 12:05:13 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.134.13:45348
2021/05/02 12:05:19 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.134.13
2021/05/02 12:05:19 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/02 12:06:09 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.134.13:8302
2021/05/02 12:06:32 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.134.13:8301
2021/05/02 12:07:12 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.134.13:8301
2021/05/02 12:07:52 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.134.13:8301
2021/05/02 12:08:09 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.134.13:8302
2021/05/02 12:08:55 [WARN] raft: Rejecting vote request from 10.242.134.13:8300 since we have a leader: 10.242.198.2:8300
2021/05/02 12:08:55 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.134.13
2021/05/02 12:08:55 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/02 12:08:55 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 12:08:55 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/02 12:08:55 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 12:08:55 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 12:09:01 [WARN] raft: Rejecting vote request from 10.242.134.13:8300 since we have a leader: 10.242.198.2:8300
2021/05/02 12:09:01 [INFO] consul: New leader elected: dev-consul-server-0
2021/05/02 12:09:02 [DEBUG] raft-net: 10.242.37.206:8300 accepted connection from: 10.242.198.2:33132
2021/05/02 12:09:03 [INFO] serf: EventMemberUpdate: dev-consul-server-1
2021/05/02 12:09:03 [INFO] serf: EventMemberUpdate: dev-consul-server-1.dc1
2021/05/02 12:15:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:35456->10.242.134.13:8301: i/o timeout
2021/05/02 12:15:39 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:15:42 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/02 12:18:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:37810->10.242.134.13:8301: i/o timeout
2021/05/02 12:18:45 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:22:12 [ERR] yamux: keepalive failed: session shutdown
2021/05/02 12:24:46 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 12:24:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:42462->10.242.134.13:8301: i/o timeout
2021/05/02 12:24:52 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:25:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:42754->10.242.134.13:8301: i/o timeout
2021/05/02 12:25:15 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:25:17 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 12:25:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:42888->10.242.134.13:8301: i/o timeout
2021/05/02 12:25:25 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:25:27 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 12:25:27 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 12:25:27 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 12:25:28 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 12:25:28 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 12:26:03 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 12:26:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:43902->10.242.134.13:8301: i/o timeout
2021/05/02 12:26:44 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:26:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:43924->10.242.134.13:8301: i/o timeout
2021/05/02 12:26:46 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:26:48 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 12:26:48 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 12:26:48 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 12:26:50 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 12:26:50 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 12:26:50 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 12:26:50 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/02 12:27:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:44420->10.242.134.13:8301: i/o timeout
2021/05/02 12:27:25 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:27:27 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 12:27:27 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 12:27:27 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 12:27:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:44438->10.242.134.13:8301: i/o timeout
2021/05/02 12:27:28 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:27:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:33580->10.242.134.13:8302: i/o timeout
2021/05/02 12:27:33 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 12:27:35 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 12:27:35 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 12:28:02 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 12:28:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:44932->10.242.134.13:8301: i/o timeout
2021/05/02 12:28:05 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:28:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:44962->10.242.134.13:8301: i/o timeout
2021/05/02 12:28:08 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:28:09 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 12:28:09 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 12:28:09 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 12:28:11 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 12:28:11 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 12:28:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:45472->10.242.134.13:8301: i/o timeout
2021/05/02 12:28:46 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:28:48 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/02 12:29:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:45944->10.242.134.13:8301: i/o timeout
2021/05/02 12:29:22 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:29:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:45992->10.242.134.13:8301: i/o timeout
2021/05/02 12:29:26 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:29:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:46028->10.242.134.13:8301: i/o timeout
2021/05/02 12:29:29 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:29:30 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 12:29:30 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 12:29:35 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 12:29:35 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 12:30:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:46432->10.242.134.13:8301: i/o timeout
2021/05/02 12:30:01 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:30:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:46504->10.242.134.13:8301: i/o timeout
2021/05/02 12:30:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:46550->10.242.134.13:8301: i/o timeout
2021/05/02 12:30:10 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:30:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:46696->10.242.134.13:8301: i/o timeout
2021/05/02 12:30:21 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:30:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:46762->10.242.134.13:8301: i/o timeout
2021/05/02 12:30:26 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:30:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:46850->10.242.134.13:8301: i/o timeout
2021/05/02 12:30:33 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:30:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:46890->10.242.134.13:8301: i/o timeout
2021/05/02 12:30:36 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:30:37 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 12:30:37 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 12:30:37 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 12:30:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:35984->10.242.134.13:8302: i/o timeout
2021/05/02 12:30:38 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 12:30:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:36112->10.242.134.13:8302: i/o timeout
2021/05/02 12:30:48 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 12:30:59 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 12:30:59 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 12:31:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:47500->10.242.134.13:8301: i/o timeout
2021/05/02 12:31:23 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:31:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:47600->10.242.134.13:8301: i/o timeout
2021/05/02 12:32:09 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/02 12:32:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:48146->10.242.134.13:8301: i/o timeout
2021/05/02 12:32:14 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:32:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:48172->10.242.134.13:8301: i/o timeout
2021/05/02 12:32:17 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:32:18 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/02 12:32:18 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.134.13
2021/05/02 12:32:18 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 12:32:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:48218->10.242.134.13:8301: i/o timeout
2021/05/02 12:32:19 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/02 12:32:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:37330->10.242.134.13:8302: i/o timeout
2021/05/02 12:32:23 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/02 12:32:32 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.134.13:8301
2021/05/02 12:32:33 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/02 12:32:37 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.134.13
2021/05/02 12:32:37 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.134.13:8300) (DC: dc1)
2021/05/02 12:32:37 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/02 12:33:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.37.206:48920->10.242.134.13:8301: i/o timeout
2021/05/02 12:33:14 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received