By type
+ CONSUL_BIND= + [ -n ] + CONSUL_CLIENT= + [ -n ] + CONSUL_DATA_DIR=/consul/data + CONSUL_CONFIG_DIR=/consul/config + [ -n ] + [ a = - ] + [ agent = agent ] + shift + set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui + [ consul = consul ] + stat -c %u /consul/data + id -u consul + [ 100 != 100 ] + stat -c %u /consul/config + id -u consul + [ 100 != 100 ] + [ ! -z ] + exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui bootstrap_expect > 0: expecting 3 servers ==> Starting Consul agent... ==> Consul agent running! Version: 'v1.0.6' Node ID: '3454d0fb-1017-7917-b868-ce15c7ffda2e' Node name: 'dev-consul-server-1' Datacenter: 'dc1' (Segment: '
') Server: true (Bootstrap: false) Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600) Cluster Addr: 10.242.254.59 (LAN: 8301, WAN: 8302) Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false ==> Log data will now stream in as it occurs: 2021/04/08 15:19:58 [INFO] raft: Initial configuration (index=0): [] 2021/04/08 15:19:58 [INFO] raft: Node at 10.242.254.59:8300 [Follower] entering Follower state (Leader: "") 2021/04/08 15:19:58 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.254.59 2021/04/08 15:19:58 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.254.59 2021/04/08 15:19:58 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan" 2021/04/08 15:19:58 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.254.59:8300) (DC: dc1) 2021/04/08 15:19:58 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp) 2021/04/08 15:19:58 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp) 2021/04/08 15:19:58 [INFO] agent: Started HTTP server on [::]:8500 (tcp) 2021/04/08 15:19:58 [INFO] agent: started state syncer 2021/04/08 15:19:58 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer 2021/04/08 15:19:58 [INFO] agent: Joining LAN cluster... 2021/04/08 15:19:58 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server] 2021/04/08 15:19:58 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:19:58 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.139.247 2021/04/08 15:19:58 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.12.17 2021/04/08 15:19:58 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:19:59 [INFO] consul: Existing Raft peers reported by dev-consul-server-0, disabling bootstrap mode 2021/04/08 15:19:59 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.12.17:8300) (DC: dc1) 2021/04/08 15:19:59 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/04/08 15:19:59 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.139.247 2021/04/08 15:19:59 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.12.17 2021/04/08 15:19:59 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan" 2021/04/08 15:19:59 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan" 2021/04/08 15:19:59 [DEBUG] raft-net: 10.242.254.59:8300 accepted connection from: 10.242.12.17:58774 2021/04/08 15:19:59 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host 2021/04/08 15:19:59 [WARN] raft: Failed to get previous log: 1091 log not found (last: 0) 2021/04/08 15:19:59 [INFO] agent: (LAN) joined: 2 Err: 2021/04/08 15:19:59 [INFO] agent: Join LAN completed. Synced with 2 initial agents 2021/04/08 15:19:59 [INFO] agent: Synced node info 2021/04/08 15:20:00 [DEBUG] raft-net: 10.242.254.59:8300 accepted connection from: 10.242.12.17:58784 ==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=431b1518-bb35-e8d8-255f-99f8f345b104&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 2021/04/08 15:20:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:58102->10.242.139.195:8301: i/o timeout 2021/04/08 15:20:59 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:21:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:45108->10.242.139.247:8301: i/o timeout 2021/04/08 15:21:14 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:21:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:45112->10.242.139.247:8301: i/o timeout 2021/04/08 15:21:17 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:21:19 [INFO] memberlist: Marking dev-consul-7467bf7b46-w5szm as failed, suspect timeout reached (2 peer confirmations) 2021/04/08 15:21:19 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:21:21 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:21:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:58236->10.242.139.195:8301: i/o timeout 2021/04/08 15:21:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:22:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:45392->10.242.139.247:8301: i/o timeout 2021/04/08 15:22:21 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:22:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:58458->10.242.139.195:8301: i/o timeout 2021/04/08 15:22:25 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:22:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:45422->10.242.139.247:8301: i/o timeout 2021/04/08 15:22:27 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:22:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:58476->10.242.139.195:8301: i/o timeout 2021/04/08 15:22:28 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:22:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:45434->10.242.139.247:8301: i/o timeout 2021/04/08 15:22:30 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:22:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:58488->10.242.139.195:8301: i/o timeout 2021/04/08 15:22:32 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:22:34 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-w5szm but other probes failed, network may be misconfigured 2021/04/08 15:22:35 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured 2021/04/08 15:22:36 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1) 2021/04/08 15:22:36 [DEBUG] raft-net: 10.242.254.59:8300 accepted connection from: 10.242.139.247:49592 2021/04/08 15:22:36 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0) 2021/04/08 15:22:36 [WARN] raft: Rejecting vote request from 10.242.139.247:8300 since we have a leader: 10.242.12.17:8300 2021/04/08 15:22:36 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured 2021/04/08 15:22:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:58594->10.242.139.195:8301: i/o timeout 2021/04/08 15:22:42 [INFO] consul: New leader elected: dev-consul-server-2 2021/04/08 15:22:43 [DEBUG] raft-net: 10.242.254.59:8300 accepted connection from: 10.242.12.17:33360 2021/04/08 15:22:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:45606->10.242.139.247:8301: i/o timeout 2021/04/08 15:22:47 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:22:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:58676->10.242.139.195:8301: i/o timeout 2021/04/08 15:22:53 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:22:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:58686->10.242.139.195:8301: i/o timeout 2021/04/08 15:22:56 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:23:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:58706->10.242.139.195:8301: i/o timeout 2021/04/08 15:23:00 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:23:04 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-w5szm but other probes failed, network may be misconfigured 2021/04/08 15:23:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:58730->10.242.139.195:8301: i/o timeout 2021/04/08 15:23:06 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:23:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:58784->10.242.139.195:8301: i/o timeout 2021/04/08 15:23:20 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:23:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:58788->10.242.139.195:8301: i/o timeout 2021/04/08 15:23:23 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:23:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:58818->10.242.139.195:8301: i/o timeout 2021/04/08 15:23:29 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:23:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:58920->10.242.139.195:8301: i/o timeout 2021/04/08 15:23:56 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:24:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:58978->10.242.139.195:8301: i/o timeout 2021/04/08 15:24:08 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:24:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:58984->10.242.139.195:8301: i/o timeout 2021/04/08 15:24:11 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:24:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:59002->10.242.139.195:8301: i/o timeout 2021/04/08 15:24:14 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:24:14 [INFO] memberlist: Marking dev-consul-7467bf7b46-w5szm as failed, suspect timeout reached (2 peer confirmations) 2021/04/08 15:24:14 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:24:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:45958->10.242.139.247:8301: i/o timeout 2021/04/08 15:24:17 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:24:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:45976->10.242.139.247:8301: i/o timeout 2021/04/08 15:24:19 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:24:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:45986->10.242.139.247:8301: i/o timeout 2021/04/08 15:24:21 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:24:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:45988->10.242.139.247:8301: i/o timeout 2021/04/08 15:24:23 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:24:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:46010->10.242.139.247:8301: i/o timeout 2021/04/08 15:24:26 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:24:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:52858->10.242.139.247:8302: i/o timeout 2021/04/08 15:24:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:46024->10.242.139.247:8301: i/o timeout 2021/04/08 15:24:28 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:24:29 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations) 2021/04/08 15:24:29 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.139.247 2021/04/08 15:24:29 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:24:35 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.139.247 2021/04/08 15:24:35 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:24:35 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2) 2021/04/08 15:24:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:46064->10.242.139.247:8301: i/o timeout 2021/04/08 15:24:37 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:24:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:46070->10.242.139.247:8301: i/o timeout 2021/04/08 15:24:39 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:24:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:46080->10.242.139.247:8301: i/o timeout 2021/04/08 15:24:41 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:24:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:52926->10.242.139.247:8302: i/o timeout 2021/04/08 15:24:43 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:24:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:46094->10.242.139.247:8301: i/o timeout 2021/04/08 15:24:43 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:24:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:46106->10.242.139.247:8301: i/o timeout 2021/04/08 15:24:46 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:24:47 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 15:24:47 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.139.195:40776 2021/04/08 15:24:47 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations) 2021/04/08 15:24:47 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.139.247 2021/04/08 15:24:47 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:24:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:46120->10.242.139.247:8301: i/o timeout 2021/04/08 15:24:48 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:24:52 [ERR] memberlist: failed to receive: read tcp 10.242.254.59:8301->10.242.139.195:46874: i/o timeout from=10.242.139.195:46874 2021/04/08 15:24:52 [ERR] memberlist: Failed to send error: write tcp 10.242.254.59:8301->10.242.139.195:46874: i/o timeout from=10.242.139.195:46874 2021/04/08 15:24:53 [WARN] memberlist: Was able to connect to dev-consul-server-0.dc1 but other probes failed, network may be misconfigured 2021/04/08 15:24:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:53000->10.242.139.247:8302: i/o timeout 2021/04/08 15:24:58 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:24:58 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.139.247:8301 2021/04/08 15:25:02 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:25:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:59262->10.242.139.195:8301: i/o timeout 2021/04/08 15:25:04 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:25:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:59268->10.242.139.195:8301: i/o timeout 2021/04/08 15:25:06 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:25:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:53074->10.242.139.247:8302: i/o timeout 2021/04/08 15:25:08 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:25:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:59296->10.242.139.195:8301: i/o timeout 2021/04/08 15:25:08 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:25:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:59308->10.242.139.195:8301: i/o timeout 2021/04/08 15:25:10 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:25:13 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/04/08 15:25:13 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.139.247 2021/04/08 15:25:13 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan" 2021/04/08 15:25:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:59326->10.242.139.195:8301: i/o timeout 2021/04/08 15:25:13 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:25:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:59340->10.242.139.195:8301: i/o timeout 2021/04/08 15:25:15 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:25:15 [INFO] memberlist: Marking dev-consul-7467bf7b46-w5szm as failed, suspect timeout reached (1 peer confirmations) 2021/04/08 15:25:15 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:25:28 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.139.247:8302 2021/04/08 15:25:29 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1) 2021/04/08 15:25:35 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.139.247 2021/04/08 15:25:35 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan" 2021/04/08 15:25:38 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-w5szm 10.242.139.195:8301 2021/04/08 15:25:47 [ERR] memberlist: failed to receive: read tcp 10.242.254.59:8301->10.242.139.195:47324: i/o timeout from=10.242.139.195:47324 2021/04/08 15:25:47 [ERR] memberlist: Failed to send error: write tcp 10.242.254.59:8301->10.242.139.195:47324: i/o timeout from=10.242.139.195:47324 2021/04/08 15:25:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:53290->10.242.139.247:8302: i/o timeout 2021/04/08 15:25:53 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:26:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:53312->10.242.139.247:8302: i/o timeout 2021/04/08 15:26:03 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:26:05 [ERR] memberlist: failed to receive: read tcp 10.242.254.59:8302->10.242.139.247:55484: i/o timeout from=10.242.139.247:55484 2021/04/08 15:26:05 [ERR] memberlist: Failed to send error: write tcp 10.242.254.59:8302->10.242.139.247:55484: i/o timeout from=10.242.139.247:55484 2021/04/08 15:26:05 [ERR] memberlist: Push/Pull with dev-consul-server-0.dc1 failed: read tcp 10.242.254.59:53226->10.242.139.247:8302: i/o timeout 2021/04/08 15:26:11 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 15:26:11 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.139.247:52114 2021/04/08 15:26:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:53388->10.242.139.247:8302: i/o timeout 2021/04/08 15:26:18 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:26:18 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-w5szm 10.242.139.195:8301 2021/04/08 15:26:23 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/04/08 15:26:23 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.139.247 2021/04/08 15:26:23 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan" 2021/04/08 15:26:29 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.139.247 2021/04/08 15:26:29 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:26:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:46604->10.242.139.247:8301: i/o timeout 2021/04/08 15:26:30 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:26:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:46610->10.242.139.247:8301: i/o timeout 2021/04/08 15:26:32 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:26:34 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (0 peer confirmations) 2021/04/08 15:26:34 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.139.247 2021/04/08 15:26:34 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:26:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:46624->10.242.139.247:8301: i/o timeout 2021/04/08 15:26:34 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:26:35 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.139.247:8302 2021/04/08 15:26:39 [ERR] memberlist: failed to receive: read tcp 10.242.254.59:8301->10.242.139.195:47708: i/o timeout from=10.242.139.195:47708 2021/04/08 15:26:39 [ERR] memberlist: Failed to send error: write tcp 10.242.254.59:8301->10.242.139.195:47708: i/o timeout from=10.242.139.195:47708 2021/04/08 15:26:58 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-w5szm 10.242.139.195:8301 2021/04/08 15:27:01 [WARN] raft: Rejecting vote request from 10.242.139.247:8300 since we have a leader: 10.242.12.17:8300 2021/04/08 15:27:38 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-w5szm 10.242.139.195:8301 2021/04/08 15:27:40 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 15:27:40 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.139.195:42686 2021/04/08 15:27:52 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 15:28:18 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-w5szm 10.242.139.195:8301 2021/04/08 15:28:58 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-w5szm 10.242.139.195:8301 2021/04/08 15:29:35 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.139.247:8302 2021/04/08 15:29:35 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.139.247 2021/04/08 15:29:35 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:29:35 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:29:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:47604->10.242.139.247:8301: i/o timeout 2021/04/08 15:29:37 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:29:39 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (0 peer confirmations) 2021/04/08 15:29:39 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.139.247 2021/04/08 15:29:39 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:29:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:47616->10.242.139.247:8301: i/o timeout 2021/04/08 15:29:40 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:29:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:60674->10.242.139.195:8301: i/o timeout 2021/04/08 15:29:43 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:29:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:60692->10.242.139.195:8301: i/o timeout 2021/04/08 15:29:46 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:29:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:60714->10.242.139.195:8301: i/o timeout 2021/04/08 15:29:49 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:29:51 [INFO] memberlist: Marking dev-consul-7467bf7b46-w5szm as failed, suspect timeout reached (1 peer confirmations) 2021/04/08 15:29:51 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:29:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:60722->10.242.139.195:8301: i/o timeout 2021/04/08 15:29:51 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:30:03 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.139.247 2021/04/08 15:30:03 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:30:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:47746->10.242.139.247:8301: i/o timeout 2021/04/08 15:30:04 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:30:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:47762->10.242.139.247:8301: i/o timeout 2021/04/08 15:30:07 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:30:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:47770->10.242.139.247:8301: i/o timeout 2021/04/08 15:30:09 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:30:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:47782->10.242.139.247:8301: i/o timeout 2021/04/08 15:30:12 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:30:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:47798->10.242.139.247:8301: i/o timeout 2021/04/08 15:30:15 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:30:15 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations) 2021/04/08 15:30:15 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.139.247 2021/04/08 15:30:15 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:30:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:47802->10.242.139.247:8301: i/o timeout 2021/04/08 15:30:17 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:30:30 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:30:30 [WARN] memberlist: Refuting a dead message (from: dev-consul-7467bf7b46-w5szm) 2021/04/08 15:30:31 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.139.247 2021/04/08 15:30:31 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan" 2021/04/08 15:30:32 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-w5szm but other probes failed, network may be misconfigured 2021/04/08 15:30:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:60938->10.242.139.195:8301: i/o timeout 2021/04/08 15:30:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:30:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:60952->10.242.139.195:8301: i/o timeout 2021/04/08 15:30:37 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:30:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:60964->10.242.139.195:8301: i/o timeout 2021/04/08 15:30:39 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:30:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:60998->10.242.139.195:8301: i/o timeout 2021/04/08 15:30:41 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:30:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:54794->10.242.139.247:8302: i/o timeout 2021/04/08 15:30:43 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:30:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:32782->10.242.139.195:8301: i/o timeout 2021/04/08 15:30:43 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:30:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:32792->10.242.139.195:8301: i/o timeout 2021/04/08 15:30:45 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:30:46 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:30:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:32796->10.242.139.195:8301: i/o timeout 2021/04/08 15:30:47 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:30:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:54846->10.242.139.247:8302: i/o timeout 2021/04/08 15:30:53 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:30:57 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.139.247 2021/04/08 15:30:57 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:30:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48038->10.242.139.247:8301: i/o timeout 2021/04/08 15:30:59 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:31:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48046->10.242.139.247:8301: i/o timeout 2021/04/08 15:31:01 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:31:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48048->10.242.139.247:8301: i/o timeout 2021/04/08 15:31:03 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:31:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48068->10.242.139.247:8301: i/o timeout 2021/04/08 15:31:05 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:31:07 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/04/08 15:31:07 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.139.247 2021/04/08 15:31:07 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan" 2021/04/08 15:31:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:54918->10.242.139.247:8302: i/o timeout 2021/04/08 15:31:08 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:31:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48090->10.242.139.247:8301: i/o timeout 2021/04/08 15:31:08 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:31:08 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-w5szm 10.242.139.195:8301 2021/04/08 15:31:11 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 15:31:11 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.139.195:43352 2021/04/08 15:31:11 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations) 2021/04/08 15:31:11 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.139.247 2021/04/08 15:31:11 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:31:11 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:31:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48102->10.242.139.247:8301: i/o timeout 2021/04/08 15:31:11 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:31:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:32924->10.242.139.195:8301: i/o timeout 2021/04/08 15:31:13 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:31:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:32942->10.242.139.195:8301: i/o timeout 2021/04/08 15:31:16 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:31:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:32952->10.242.139.195:8301: i/o timeout 2021/04/08 15:31:18 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:31:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:32970->10.242.139.195:8301: i/o timeout 2021/04/08 15:31:21 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:31:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:32972->10.242.139.195:8301: i/o timeout 2021/04/08 15:31:23 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:31:24 [INFO] memberlist: Marking dev-consul-7467bf7b46-w5szm as failed, suspect timeout reached (1 peer confirmations) 2021/04/08 15:31:24 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:31:25 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.139.247 2021/04/08 15:31:25 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan" 2021/04/08 15:31:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:32992->10.242.139.195:8301: i/o timeout 2021/04/08 15:31:25 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:31:31 [ERR] memberlist: failed to receive: read tcp 10.242.254.59:8301->10.242.139.195:49398: i/o timeout from=10.242.139.195:49398 2021/04/08 15:31:31 [ERR] memberlist: Failed to send error: write tcp 10.242.254.59:8301->10.242.139.195:49398: i/o timeout from=10.242.139.195:49398 2021/04/08 15:31:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:55042->10.242.139.247:8302: i/o timeout 2021/04/08 15:31:33 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:31:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:55086->10.242.139.247:8302: i/o timeout 2021/04/08 15:31:43 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:31:48 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-w5szm 10.242.139.195:8301 2021/04/08 15:31:58 [WARN] raft: Rejecting vote request from 10.242.139.247:8300 since we have a leader: 10.242.12.17:8300 2021/04/08 15:31:58 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.139.247 2021/04/08 15:31:58 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:31:58 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:32:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:33146->10.242.139.195:8301: i/o timeout 2021/04/08 15:32:00 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:32:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:33170->10.242.139.195:8301: i/o timeout 2021/04/08 15:32:05 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:32:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:33176->10.242.139.195:8301: i/o timeout 2021/04/08 15:32:08 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:32:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48380->10.242.139.247:8301: i/o timeout 2021/04/08 15:32:11 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:32:12 [INFO] memberlist: Marking dev-consul-7467bf7b46-w5szm as failed, suspect timeout reached (1 peer confirmations) 2021/04/08 15:32:12 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:32:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48392->10.242.139.247:8301: i/o timeout 2021/04/08 15:32:13 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:32:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48404->10.242.139.247:8301: i/o timeout 2021/04/08 15:32:16 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:32:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:55252->10.242.139.247:8302: i/o timeout 2021/04/08 15:32:18 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:32:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48416->10.242.139.247:8301: i/o timeout 2021/04/08 15:32:18 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:32:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48432->10.242.139.247:8301: i/o timeout 2021/04/08 15:32:20 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:32:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48438->10.242.139.247:8301: i/o timeout 2021/04/08 15:32:22 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:32:22 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations) 2021/04/08 15:32:22 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.139.247 2021/04/08 15:32:22 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:32:28 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.139.247:8301 2021/04/08 15:32:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:55280->10.242.139.247:8302: i/o timeout 2021/04/08 15:32:28 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:32:37 [WARN] raft: Rejecting vote request from 10.242.139.247:8300 since we have a leader: 10.242.12.17:8300 2021/04/08 15:32:38 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 15:32:38 [WARN] memberlist: Was able to connect to dev-consul-server-0.dc1 but other probes failed, network may be misconfigured 2021/04/08 15:32:47 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/04/08 15:32:47 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.139.247 2021/04/08 15:32:47 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan" 2021/04/08 15:32:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:55392->10.242.139.247:8302: i/o timeout 2021/04/08 15:32:48 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:33:08 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-w5szm 10.242.139.195:8301 2021/04/08 15:33:35 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.139.247:8302 2021/04/08 15:33:37 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1) 2021/04/08 15:33:41 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.139.247 2021/04/08 15:33:41 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan" 2021/04/08 15:33:41 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.139.247 2021/04/08 15:33:41 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:33:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48794->10.242.139.247:8301: i/o timeout 2021/04/08 15:33:43 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:33:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48802->10.242.139.247:8301: i/o timeout 2021/04/08 15:33:45 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:33:47 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (0 peer confirmations) 2021/04/08 15:33:47 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.139.247 2021/04/08 15:33:47 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:33:48 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.139.247:8301 2021/04/08 15:33:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:55656->10.242.139.247:8302: i/o timeout 2021/04/08 15:33:48 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:33:58 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 15:33:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:55680->10.242.139.247:8302: i/o timeout 2021/04/08 15:33:58 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:34:01 [WARN] raft: Rejecting vote request from 10.242.139.247:8300 since we have a leader: 10.242.12.17:8300 2021/04/08 15:34:01 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.139.247 2021/04/08 15:34:01 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:34:01 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2) 2021/04/08 15:34:03 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:34:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48896->10.242.139.247:8301: i/o timeout 2021/04/08 15:34:03 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:34:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:33726->10.242.139.195:8301: i/o timeout 2021/04/08 15:34:06 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:34:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:33746->10.242.139.195:8301: i/o timeout 2021/04/08 15:34:09 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:34:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:33762->10.242.139.195:8301: i/o timeout 2021/04/08 15:34:13 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:34:16 [INFO] consul: New leader elected: dev-consul-server-2 2021/04/08 15:34:16 [DEBUG] raft-net: 10.242.254.59:8300 accepted connection from: 10.242.12.17:45178 2021/04/08 15:34:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48952->10.242.139.247:8301: i/o timeout 2021/04/08 15:34:16 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:34:18 [INFO] memberlist: Marking dev-consul-7467bf7b46-w5szm as failed, suspect timeout reached (1 peer confirmations) 2021/04/08 15:34:18 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:34:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:33784->10.242.139.195:8301: i/o timeout 2021/04/08 15:34:19 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:34:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48984->10.242.139.247:8301: i/o timeout 2021/04/08 15:34:22 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:34:23 [WARN] memberlist: Was able to connect to dev-consul-server-0.dc1 but other probes failed, network may be misconfigured 2021/04/08 15:34:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49006->10.242.139.247:8301: i/o timeout 2021/04/08 15:34:25 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:34:26 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations) 2021/04/08 15:34:26 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.139.247 2021/04/08 15:34:26 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:34:27 [WARN] raft: Rejecting vote request from 10.242.139.247:8300 since we have a leader: 10.242.12.17:8300 2021/04/08 15:34:27 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.139.247 2021/04/08 15:34:27 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:34:27 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured 2021/04/08 15:34:28 [WARN] memberlist: Was able to connect to dev-consul-server-0.dc1 but other probes failed, network may be misconfigured 2021/04/08 15:34:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49028->10.242.139.247:8301: i/o timeout 2021/04/08 15:34:29 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:34:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49038->10.242.139.247:8301: i/o timeout 2021/04/08 15:34:32 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:34:33 [INFO] consul: New leader elected: dev-consul-server-2 2021/04/08 15:34:33 [DEBUG] raft-net: 10.242.254.59:8300 accepted connection from: 10.242.12.17:45460 2021/04/08 15:34:35 [WARN] raft: Rejecting vote request from 10.242.139.247:8300 since we have a leader: 10.242.12.17:8300 2021/04/08 15:34:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49056->10.242.139.247:8301: i/o timeout 2021/04/08 15:34:35 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:34:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49068->10.242.139.247:8301: i/o timeout 2021/04/08 15:34:37 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:34:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49080->10.242.139.247:8301: i/o timeout 2021/04/08 15:34:39 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:34:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49088->10.242.139.247:8301: i/o timeout 2021/04/08 15:34:42 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:34:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49108->10.242.139.247:8301: i/o timeout 2021/04/08 15:34:45 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:34:47 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations) 2021/04/08 15:34:47 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.139.247 2021/04/08 15:34:47 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:34:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49112->10.242.139.247:8301: i/o timeout 2021/04/08 15:34:47 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:34:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:55960->10.242.139.247:8302: i/o timeout 2021/04/08 15:34:48 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:34:50 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:34:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:33954->10.242.139.195:8301: i/o timeout 2021/04/08 15:34:51 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:34:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:33976->10.242.139.195:8301: i/o timeout 2021/04/08 15:34:55 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:34:57 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1) 2021/04/08 15:34:57 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.139.247 2021/04/08 15:34:57 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:34:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:33988->10.242.139.195:8301: i/o timeout 2021/04/08 15:34:57 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:35:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49218->10.242.139.247:8301: i/o timeout 2021/04/08 15:35:07 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:35:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:34054->10.242.139.195:8301: i/o timeout 2021/04/08 15:35:11 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:35:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:34076->10.242.139.195:8301: i/o timeout 2021/04/08 15:35:16 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:35:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49284->10.242.139.247:8301: i/o timeout 2021/04/08 15:35:20 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:35:22 [INFO] memberlist: Marking dev-consul-7467bf7b46-w5szm as failed, suspect timeout reached (2 peer confirmations) 2021/04/08 15:35:22 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:35:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:34106->10.242.139.195:8301: i/o timeout 2021/04/08 15:35:22 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:35:23 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:35:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:34128->10.242.139.195:8301: i/o timeout 2021/04/08 15:35:26 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:35:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49326->10.242.139.247:8301: i/o timeout 2021/04/08 15:35:29 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:35:30 [ERR] memberlist: failed to receive: read tcp 10.242.254.59:8301->10.242.139.195:51266: i/o timeout from=10.242.139.195:51266 2021/04/08 15:35:30 [ERR] memberlist: Failed to send error: write tcp 10.242.254.59:8301->10.242.139.195:51266: i/o timeout from=10.242.139.195:51266 2021/04/08 15:35:30 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:35:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:34152->10.242.139.195:8301: i/o timeout 2021/04/08 15:35:31 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:35:35 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:35:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49378->10.242.139.247:8301: i/o timeout 2021/04/08 15:35:40 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:35:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49404->10.242.139.247:8301: i/o timeout 2021/04/08 15:35:46 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:35:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:34238->10.242.139.195:8301: i/o timeout 2021/04/08 15:35:49 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:35:52 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:35:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:34250->10.242.139.195:8301: i/o timeout 2021/04/08 15:35:52 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:35:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49452->10.242.139.247:8301: i/o timeout 2021/04/08 15:35:56 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:35:58 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-w5szm 10.242.139.195:8301 2021/04/08 15:35:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49466->10.242.139.247:8301: i/o timeout 2021/04/08 15:35:58 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:36:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49478->10.242.139.247:8301: i/o timeout 2021/04/08 15:36:00 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:36:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49492->10.242.139.247:8301: i/o timeout 2021/04/08 15:36:03 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:36:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49500->10.242.139.247:8301: i/o timeout 2021/04/08 15:36:05 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:36:06 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:36:08 [WARN] raft: Rejecting vote request from 10.242.139.247:8300 since we have a leader: 10.242.12.17:8300 2021/04/08 15:36:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:34350->10.242.139.195:8301: i/o timeout 2021/04/08 15:36:11 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:36:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49536->10.242.139.247:8301: i/o timeout 2021/04/08 15:36:14 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:36:15 [INFO] consul: New leader elected: dev-consul-server-2 2021/04/08 15:36:16 [DEBUG] raft-net: 10.242.254.59:8300 accepted connection from: 10.242.12.17:47328 2021/04/08 15:36:17 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-w5szm but other probes failed, network may be misconfigured 2021/04/08 15:36:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49572->10.242.139.247:8301: i/o timeout 2021/04/08 15:36:19 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:36:21 [INFO] memberlist: Marking dev-consul-7467bf7b46-w5szm as failed, suspect timeout reached (2 peer confirmations) 2021/04/08 15:36:21 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:36:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49578->10.242.139.247:8301: i/o timeout 2021/04/08 15:36:21 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:36:23 [WARN] memberlist: Was able to connect to dev-consul-server-0.dc1 but other probes failed, network may be misconfigured 2021/04/08 15:36:24 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.139.247 2021/04/08 15:36:24 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:36:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49600->10.242.139.247:8301: i/o timeout 2021/04/08 15:36:24 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:36:27 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0) 2021/04/08 15:36:27 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.139.247 2021/04/08 15:36:27 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:36:28 [WARN] memberlist: Was able to connect to dev-consul-server-0.dc1 but other probes failed, network may be misconfigured 2021/04/08 15:36:29 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured 2021/04/08 15:36:32 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured 2021/04/08 15:36:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49648->10.242.139.247:8301: i/o timeout 2021/04/08 15:36:34 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:36:35 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1) 2021/04/08 15:36:36 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-w5szm 10.242.139.195:8301 2021/04/08 15:36:38 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:36:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:34510->10.242.139.195:8301: i/o timeout 2021/04/08 15:36:43 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:36:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:34520->10.242.139.195:8301: i/o timeout 2021/04/08 15:36:46 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:36:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49714->10.242.139.247:8301: i/o timeout 2021/04/08 15:36:49 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:36:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:34546->10.242.139.195:8301: i/o timeout 2021/04/08 15:36:52 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:36:53 [WARN] memberlist: Was able to connect to dev-consul-server-0.dc1 but other probes failed, network may be misconfigured 2021/04/08 15:36:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49748->10.242.139.247:8301: i/o timeout 2021/04/08 15:36:55 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:36:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:34572->10.242.139.195:8301: i/o timeout 2021/04/08 15:36:57 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:36:58 [WARN] memberlist: Was able to connect to dev-consul-server-0.dc1 but other probes failed, network may be misconfigured 2021/04/08 15:36:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49776->10.242.139.247:8301: i/o timeout 2021/04/08 15:36:59 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:37:00 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations) 2021/04/08 15:37:00 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.139.247 2021/04/08 15:37:00 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:37:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:34602->10.242.139.195:8301: i/o timeout 2021/04/08 15:37:01 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:37:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:34614->10.242.139.195:8301: i/o timeout 2021/04/08 15:37:03 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:37:04 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:37:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:34622->10.242.139.195:8301: i/o timeout 2021/04/08 15:37:05 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:37:08 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.139.247:8301 2021/04/08 15:37:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:56678->10.242.139.247:8302: i/o timeout 2021/04/08 15:37:16 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1) 2021/04/08 15:37:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:56766->10.242.139.247:8302: i/o timeout 2021/04/08 15:37:33 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:37:42 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.139.247 2021/04/08 15:37:42 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:37:43 [WARN] memberlist: Was able to connect to dev-consul-server-0.dc1 but other probes failed, network may be misconfigured 2021/04/08 15:37:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49986->10.242.139.247:8301: i/o timeout 2021/04/08 15:37:45 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:37:47 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (0 peer confirmations) 2021/04/08 15:37:47 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.139.247 2021/04/08 15:37:47 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:37:47 [WARN] raft: Rejecting vote request from 10.242.139.247:8300 since we have a leader: 10.242.12.17:8300 2021/04/08 15:37:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:50000->10.242.139.247:8301: i/o timeout 2021/04/08 15:37:47 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:37:48 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.139.247:8301 2021/04/08 15:37:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:56836->10.242.139.247:8302: i/o timeout 2021/04/08 15:37:48 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:37:53 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.139.247 2021/04/08 15:37:53 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:37:54 [ERR] memberlist: failed to receive: read tcp 10.242.254.59:8301->10.242.139.195:52882: i/o timeout from=10.242.139.195:52882 2021/04/08 15:37:54 [ERR] memberlist: Failed to send error: write tcp 10.242.254.59:8301->10.242.139.195:52882: i/o timeout from=10.242.139.195:52882 2021/04/08 15:38:02 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:38:02 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-w5szm) 2021/04/08 15:38:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:50088->10.242.139.247:8301: i/o timeout 2021/04/08 15:38:07 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:38:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:34914->10.242.139.195:8301: i/o timeout 2021/04/08 15:38:10 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:38:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:56952->10.242.139.247:8302: i/o timeout 2021/04/08 15:38:13 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:38:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:50106->10.242.139.247:8301: i/o timeout 2021/04/08 15:38:13 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:38:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:34942->10.242.139.195:8301: i/o timeout 2021/04/08 15:38:16 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:38:19 [INFO] memberlist: Marking dev-consul-7467bf7b46-w5szm as failed, suspect timeout reached (1 peer confirmations) 2021/04/08 15:38:19 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:38:19 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations) 2021/04/08 15:38:19 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.139.247 2021/04/08 15:38:19 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:38:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:34960->10.242.139.195:8301: i/o timeout 2021/04/08 15:38:20 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:38:23 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.139.247:8301 2021/04/08 15:38:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:56998->10.242.139.247:8302: i/o timeout 2021/04/08 15:38:23 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:38:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:57046->10.242.139.247:8302: i/o timeout 2021/04/08 15:38:33 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:38:36 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 15:38:36 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.139.195:47748 2021/04/08 15:38:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:57090->10.242.139.247:8302: i/o timeout 2021/04/08 15:38:43 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:38:43 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/04/08 15:38:43 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.139.247 2021/04/08 15:38:43 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan" 2021/04/08 15:39:03 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.139.247:8301 2021/04/08 15:39:06 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.139.247 2021/04/08 15:39:06 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan" 2021/04/08 15:39:12 [WARN] raft: Rejecting vote request from 10.242.139.247:8300 since we have a leader: 10.242.12.17:8300 2021/04/08 15:39:12 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.139.247 2021/04/08 15:39:12 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:39:20 [WARN] raft: Heartbeat timeout from "10.242.12.17:8300" reached, starting election 2021/04/08 15:39:20 [INFO] raft: Node at 10.242.254.59:8300 [Candidate] entering Candidate state in term 21 2021/04/08 15:39:20 [INFO] raft: Election won. Tally: 2 2021/04/08 15:39:20 [INFO] raft: Node at 10.242.254.59:8300 [Leader] entering Leader state 2021/04/08 15:39:20 [INFO] raft: Added peer 0e6d1abb-47a7-4121-d0d1-64118dab1223, starting replication 2021/04/08 15:39:20 [INFO] raft: Added peer c52c5261-fd75-2dd5-3bad-46a881682908, starting replication 2021/04/08 15:39:20 [INFO] consul: cluster leadership acquired 2021/04/08 15:39:20 [INFO] consul: New leader elected: dev-consul-server-1 2021/04/08 15:39:20 [INFO] raft: pipelining replication to peer {Voter 0e6d1abb-47a7-4121-d0d1-64118dab1223 10.242.12.17:8300} 2021/04/08 15:39:20 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive 2021/04/08 15:39:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:50412->10.242.139.247:8301: i/o timeout 2021/04/08 15:39:20 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:39:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:50424->10.242.139.247:8301: i/o timeout 2021/04/08 15:39:22 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:39:23 [WARN] raft: Failed to contact c52c5261-fd75-2dd5-3bad-46a881682908 in 2.518816476s 2021/04/08 15:39:23 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:39:24 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (0 peer confirmations) 2021/04/08 15:39:24 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.139.247 2021/04/08 15:39:24 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:39:24 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical 2021/04/08 15:39:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:39:25 [WARN] raft: Failed to contact c52c5261-fd75-2dd5-3bad-46a881682908 in 4.970237033s 2021/04/08 15:39:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:39:28 [WARN] raft: Failed to contact c52c5261-fd75-2dd5-3bad-46a881682908 in 7.390343116s 2021/04/08 15:39:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:39:29 [WARN] raft: Rejecting vote request from 10.242.139.247:8300 since we have a leader: 10.242.254.59:8300 2021/04/08 15:39:29 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:39:30 [ERR] raft: Failed to make RequestVote RPC to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: read tcp 10.242.254.59:56904->10.242.139.247:8300: i/o timeout 2021/04/08 15:39:30 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: read tcp 10.242.254.59:56906->10.242.139.247:8300: i/o timeout 2021/04/08 15:39:30 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:39:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:39:31 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:39:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:39:32 [WARN] raft: Failed to contact c52c5261-fd75-2dd5-3bad-46a881682908 in 2.500240293s 2021/04/08 15:39:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:39:35 [WARN] raft: Failed to contact c52c5261-fd75-2dd5-3bad-46a881682908 in 4.929580846s 2021/04/08 15:39:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:39:37 [WARN] raft: Failed to contact c52c5261-fd75-2dd5-3bad-46a881682908 in 7.385922749s 2021/04/08 15:39:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:39:40 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: read tcp 10.242.254.59:56916->10.242.139.247:8300: i/o timeout 2021/04/08 15:39:40 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:39:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:39:41 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: read tcp 10.242.254.59:56962->10.242.139.247:8300: i/o timeout 2021/04/08 15:39:41 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:39:42 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-w5szm 10.242.139.195:8301 2021/04/08 15:39:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:39:43 [WARN] raft: Rejecting vote request from 10.242.139.247:8300 since we have a leader: 10.242.254.59:8300 2021/04/08 15:39:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:57368->10.242.139.247:8302: i/o timeout 2021/04/08 15:39:43 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:39:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:39:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:39:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:39:50 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: read tcp 10.242.254.59:57006->10.242.139.247:8300: i/o timeout 2021/04/08 15:39:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:39:50 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:39:51 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: read tcp 10.242.254.59:57010->10.242.139.247:8300: i/o timeout 2021/04/08 15:39:52 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:39:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:39:53 [ERR] memberlist: failed to receive: read tcp 10.242.254.59:8301->10.242.139.247:44896: i/o timeout from=10.242.139.247:44896 2021/04/08 15:39:53 [ERR] memberlist: Failed to send error: write tcp 10.242.254.59:8301->10.242.139.247:44896: i/o timeout from=10.242.139.247:44896 2021/04/08 15:39:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:57394->10.242.139.247:8302: i/o timeout 2021/04/08 15:39:53 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:39:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:39:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:39:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:39:59 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.139.247 2021/04/08 15:39:59 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0) 2021/04/08 15:39:59 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:39:59 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive 2021/04/08 15:40:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:00 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: read tcp 10.242.254.59:57056->10.242.139.247:8300: i/o timeout 2021/04/08 15:40:02 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: read tcp 10.242.254.59:57068->10.242.139.247:8300: i/o timeout 2021/04/08 15:40:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:50618->10.242.139.247:8301: i/o timeout 2021/04/08 15:40:02 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:40:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:50630->10.242.139.247:8301: i/o timeout 2021/04/08 15:40:04 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:40:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:50644->10.242.139.247:8301: i/o timeout 2021/04/08 15:40:06 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:40:06 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (0 peer confirmations) 2021/04/08 15:40:06 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.139.247 2021/04/08 15:40:06 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:40:06 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical 2021/04/08 15:40:07 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/04/08 15:40:07 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.139.247 2021/04/08 15:40:07 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan" 2021/04/08 15:40:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:57492->10.242.139.247:8302: i/o timeout 2021/04/08 15:40:08 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:40:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:10 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: read tcp 10.242.254.59:57102->10.242.139.247:8300: i/o timeout 2021/04/08 15:40:10 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:40:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:13 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: read tcp 10.242.254.59:57120->10.242.139.247:8300: i/o timeout 2021/04/08 15:40:14 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:40:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:19 [WARN] raft: Rejecting vote request from 10.242.139.247:8300 since we have a leader: 10.242.254.59:8300 2021/04/08 15:40:19 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:40:19 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 15:40:20 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:40:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:20 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: read tcp 10.242.254.59:57156->10.242.139.247:8300: i/o timeout 2021/04/08 15:40:21 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:40:21 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:40:22 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.139.247:8301 2021/04/08 15:40:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:22 [WARN] raft: Failed to contact c52c5261-fd75-2dd5-3bad-46a881682908 in 2.500259161s 2021/04/08 15:40:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:25 [WARN] raft: Failed to contact c52c5261-fd75-2dd5-3bad-46a881682908 in 4.984313983s 2021/04/08 15:40:26 [ERR] memberlist: failed to receive: read tcp 10.242.254.59:8301->10.242.139.195:53840: i/o timeout from=10.242.139.195:53840 2021/04/08 15:40:26 [ERR] memberlist: Failed to send error: write tcp 10.242.254.59:8301->10.242.139.195:53840: i/o timeout from=10.242.139.195:53840 2021/04/08 15:40:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:27 [WARN] raft: Failed to contact c52c5261-fd75-2dd5-3bad-46a881682908 in 7.448184253s 2021/04/08 15:40:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:31 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: read tcp 10.242.254.59:57174->10.242.139.247:8300: i/o timeout 2021/04/08 15:40:31 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: read tcp 10.242.254.59:57202->10.242.139.247:8300: i/o timeout 2021/04/08 15:40:31 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:40:32 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:40:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:41 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.139.247:8302 2021/04/08 15:40:41 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: read tcp 10.242.254.59:57250->10.242.139.247:8300: i/o timeout 2021/04/08 15:40:42 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: read tcp 10.242.254.59:57252->10.242.139.247:8300: i/o timeout 2021/04/08 15:40:42 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:40:42 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:40:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:48 [ERR] memberlist: failed to receive: read tcp 10.242.254.59:8301->10.242.139.247:45426: i/o timeout from=10.242.139.247:45426 2021/04/08 15:40:48 [ERR] memberlist: Failed to send error: write tcp 10.242.254.59:8301->10.242.139.247:45426: i/o timeout from=10.242.139.247:45426 2021/04/08 15:40:48 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.139.247 2021/04/08 15:40:48 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:40:48 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive 2021/04/08 15:40:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:50852->10.242.139.247:8301: i/o timeout 2021/04/08 15:40:49 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:40:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:50860->10.242.139.247:8301: i/o timeout 2021/04/08 15:40:51 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:40:52 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: read tcp 10.242.254.59:57302->10.242.139.247:8300: i/o timeout 2021/04/08 15:40:52 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: read tcp 10.242.254.59:57312->10.242.139.247:8300: i/o timeout 2021/04/08 15:40:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:53 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-w5szm 10.242.139.195:8301 2021/04/08 15:40:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:50870->10.242.139.247:8301: i/o timeout 2021/04/08 15:40:53 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:40:53 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (0 peer confirmations) 2021/04/08 15:40:53 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.139.247 2021/04/08 15:40:53 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:40:53 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical 2021/04/08 15:40:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:40:55 [WARN] raft: Rejecting vote request from 10.242.139.247:8300 since we have a leader: 10.242.254.59:8300 2021/04/08 15:40:55 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.139.247 2021/04/08 15:40:55 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan" 2021/04/08 15:40:56 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:40:57 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:40:58 [WARN] raft: Failed to contact c52c5261-fd75-2dd5-3bad-46a881682908 in 2.500251021s 2021/04/08 15:40:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:00 [WARN] raft: Failed to contact c52c5261-fd75-2dd5-3bad-46a881682908 in 4.969835443s 2021/04/08 15:41:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:03 [WARN] raft: Failed to contact c52c5261-fd75-2dd5-3bad-46a881682908 in 7.456870042s 2021/04/08 15:41:03 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: read tcp 10.242.254.59:57360->10.242.139.247:8300: i/o timeout 2021/04/08 15:41:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:57756->10.242.139.247:8302: i/o timeout 2021/04/08 15:41:03 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:41:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:06 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:41:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:06 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: read tcp 10.242.254.59:57364->10.242.139.247:8300: i/o timeout 2021/04/08 15:41:07 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:41:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:57804->10.242.139.247:8302: i/o timeout 2021/04/08 15:41:13 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:41:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:15 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 15:41:15 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF 2021/04/08 15:41:16 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: read tcp 10.242.254.59:57420->10.242.139.247:8300: i/o timeout 2021/04/08 15:41:17 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: read tcp 10.242.254.59:57432->10.242.139.247:8300: i/o timeout 2021/04/08 15:41:17 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:41:18 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:41:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:21 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:41:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:57828->10.242.139.247:8302: i/o timeout 2021/04/08 15:41:23 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:41:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:28 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: read tcp 10.242.254.59:57484->10.242.139.247:8300: i/o timeout 2021/04/08 15:41:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:29 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:41:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:31 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: read tcp 10.242.254.59:57494->10.242.139.247:8300: i/o timeout 2021/04/08 15:41:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:33 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.139.247:8301 2021/04/08 15:41:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:57900->10.242.139.247:8302: i/o timeout 2021/04/08 15:41:33 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:41:33 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/04/08 15:41:33 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.139.247 2021/04/08 15:41:33 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan" 2021/04/08 15:41:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:39 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: read tcp 10.242.254.59:57536->10.242.139.247:8300: i/o timeout 2021/04/08 15:41:39 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:41:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:41 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:41:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:48 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.139.247 2021/04/08 15:41:48 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:41:48 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive 2021/04/08 15:41:48 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.139.247 2021/04/08 15:41:48 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan" 2021/04/08 15:41:48 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1) 2021/04/08 15:41:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:51140->10.242.139.247:8301: i/o timeout 2021/04/08 15:41:49 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:41:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:50 [WARN] raft: Failed to contact c52c5261-fd75-2dd5-3bad-46a881682908 in 2.500183301s 2021/04/08 15:41:51 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: read tcp 10.242.254.59:57592->10.242.139.247:8300: i/o timeout 2021/04/08 15:41:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:52 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured 2021/04/08 15:41:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:51160->10.242.139.247:8301: i/o timeout 2021/04/08 15:41:53 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:41:53 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (0 peer confirmations) 2021/04/08 15:41:53 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.139.247 2021/04/08 15:41:53 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.139.247:8300) (DC: dc1) 2021/04/08 15:41:53 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical 2021/04/08 15:41:54 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:41:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:55 [WARN] raft: Failed to contact c52c5261-fd75-2dd5-3bad-46a881682908 in 2.500187566s 2021/04/08 15:41:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:58 [WARN] raft: Failed to contact c52c5261-fd75-2dd5-3bad-46a881682908 in 4.971844373s 2021/04/08 15:41:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:41:58 [WARN] memberlist: Was able to connect to dev-consul-server-0.dc1 but other probes failed, network may be misconfigured 2021/04/08 15:42:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:00 [WARN] raft: Failed to contact c52c5261-fd75-2dd5-3bad-46a881682908 in 7.469587811s 2021/04/08 15:42:01 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:42:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:04 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: read tcp 10.242.254.59:57586->10.242.139.247:8300: i/o timeout 2021/04/08 15:42:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:05 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:42:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:58066->10.242.139.247:8302: i/o timeout 2021/04/08 15:42:08 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:42:09 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:42:09 [INFO] consul: member 'dev-consul-7467bf7b46-w5szm' joined, marking health alive 2021/04/08 15:42:10 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0) 2021/04/08 15:42:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:11 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: read tcp 10.242.254.59:57684->10.242.139.247:8300: i/o timeout 2021/04/08 15:42:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:36064->10.242.139.195:8301: i/o timeout 2021/04/08 15:42:12 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:42:13 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.139.247:8301 2021/04/08 15:42:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:36072->10.242.139.195:8301: i/o timeout 2021/04/08 15:42:14 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:42:15 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: read tcp 10.242.254.59:57704->10.242.139.247:8300: i/o timeout 2021/04/08 15:42:15 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:42:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:36098->10.242.139.195:8301: i/o timeout 2021/04/08 15:42:17 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:42:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:58116->10.242.139.247:8302: i/o timeout 2021/04/08 15:42:18 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:42:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:36104->10.242.139.195:8301: i/o timeout 2021/04/08 15:42:19 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:42:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:21 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:42:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:36120->10.242.139.195:8301: i/o timeout 2021/04/08 15:42:22 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:42:24 [INFO] memberlist: Marking dev-consul-7467bf7b46-w5szm as failed, suspect timeout reached (1 peer confirmations) 2021/04/08 15:42:24 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-w5szm 10.242.139.195 2021/04/08 15:42:24 [INFO] consul: member 'dev-consul-7467bf7b46-w5szm' failed, marking health critical 2021/04/08 15:42:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:36138->10.242.139.195:8301: i/o timeout 2021/04/08 15:42:24 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:42:25 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: read tcp 10.242.254.59:57754->10.242.139.247:8300: i/o timeout 2021/04/08 15:42:26 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:42:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:27 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 15:42:27 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF 2021/04/08 15:42:29 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:42:29 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 15:42:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:31 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: read tcp 10.242.254.59:57786->10.242.139.247:8300: i/o timeout 2021/04/08 15:42:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:58196->10.242.139.247:8302: i/o timeout 2021/04/08 15:42:33 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:42:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:36 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: read tcp 10.242.254.59:57814->10.242.139.247:8300: i/o timeout 2021/04/08 15:42:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:37 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:42:38 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.139.247 2021/04/08 15:42:38 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan" 2021/04/08 15:42:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:42 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:42:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:47 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: read tcp 10.242.254.59:57864->10.242.139.247:8300: i/o timeout 2021/04/08 15:42:48 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:42:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:52 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: read tcp 10.242.254.59:57886->10.242.139.247:8300: i/o timeout 2021/04/08 15:42:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:53 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.139.247:8301 2021/04/08 15:42:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:58 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: dial tcp ->10.242.139.247:8300: i/o timeout 2021/04/08 15:42:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:42:58 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:43:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:02 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:43:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:08 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 15:43:08 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF 2021/04/08 15:43:08 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: dial tcp ->10.242.139.247:8300: i/o timeout 2021/04/08 15:43:09 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:43:11 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:43:12 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: dial tcp ->10.242.139.247:8300: i/o timeout 2021/04/08 15:43:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:19 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: dial tcp ->10.242.139.247:8300: i/o timeout 2021/04/08 15:43:20 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.139.247:8300: i/o timeout 2021/04/08 15:43:21 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:43:21 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:43:22 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:43:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:30 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.139.247:8300: i/o timeout 2021/04/08 15:43:31 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: dial tcp ->10.242.139.247:8300: i/o timeout 2021/04/08 15:43:32 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: dial tcp ->10.242.139.247:8300: i/o timeout 2021/04/08 15:43:33 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:43:33 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-w5szm 10.242.139.195:8301 2021/04/08 15:43:33 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:43:33 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 15:43:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:42 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.139.247:8300: i/o timeout 2021/04/08 15:43:42 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:43:43 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: dial tcp ->10.242.139.247:8300: i/o timeout 2021/04/08 15:43:45 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:43:46 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:43:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:52 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: dial tcp ->10.242.139.247:8300: i/o timeout 2021/04/08 15:43:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:43:54 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.139.247:8300: i/o timeout 2021/04/08 15:43:56 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: dial tcp ->10.242.139.247:8300: i/o timeout 2021/04/08 15:43:57 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:43:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:44:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:44:02 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:44:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:44:03 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:44:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:44:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:44:06 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.139.247:8300: i/o timeout 2021/04/08 15:44:09 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:44:09 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 15:44:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:44:12 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: dial tcp ->10.242.139.247:8300: i/o timeout 2021/04/08 15:44:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:44:13 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: dial tcp ->10.242.139.247:8300: i/o timeout 2021/04/08 15:44:13 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-w5szm 10.242.139.195:8301 2021/04/08 15:44:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:44:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:44:17 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.139.200 2021/04/08 15:44:17 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.139.200:8300) (DC: dc1) 2021/04/08 15:44:17 [INFO] raft: Updating configuration with AddNonvoter (e9afffdd-6cc9-d1e4-c968-ec678a43b132, 10.242.139.200:8300) to [{Suffrage:Voter ID:0e6d1abb-47a7-4121-d0d1-64118dab1223 Address:10.242.12.17:8300} {Suffrage:Voter ID:c52c5261-fd75-2dd5-3bad-46a881682908 Address:10.242.139.247:8300} {Suffrage:Voter ID:3454d0fb-1017-7917-b868-ce15c7ffda2e Address:10.242.254.59:8300} {Suffrage:Nonvoter ID:e9afffdd-6cc9-d1e4-c968-ec678a43b132 Address:10.242.139.200:8300}] 2021/04/08 15:44:17 [INFO] raft: Added peer e9afffdd-6cc9-d1e4-c968-ec678a43b132, starting replication 2021/04/08 15:44:17 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive 2021/04/08 15:44:17 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.139.200 2021/04/08 15:44:17 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan" 2021/04/08 15:44:17 [WARN] raft: AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300} rejected, sending older logs (next: 1) 2021/04/08 15:44:18 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.139.247:8300: i/o timeout 2021/04/08 15:44:18 [INFO] raft: pipelining replication to peer {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300} 2021/04/08 15:44:20 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive 2021/04/08 15:44:23 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:44:23 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: dial tcp ->10.242.139.247:8300: getsockopt: connection refused 2021/04/08 15:44:23 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: dial tcp ->10.242.139.247:8300: getsockopt: connection refused 2021/04/08 15:44:23 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:44:25 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:44:26 [WARN] raft: Failed to contact e9afffdd-6cc9-d1e4-c968-ec678a43b132 in 2.50022415s 2021/04/08 15:44:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:44:28 [WARN] raft: Failed to contact e9afffdd-6cc9-d1e4-c968-ec678a43b132 in 4.950686309s 2021/04/08 15:44:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:44:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:44:31 [WARN] raft: Failed to contact e9afffdd-6cc9-d1e4-c968-ec678a43b132 in 7.424489292s 2021/04/08 15:44:31 [ERR] memberlist: Failed fallback ping: write tcp 10.242.254.59:50916->10.242.139.200:8301: i/o timeout 2021/04/08 15:44:31 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:44:32 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0) 2021/04/08 15:44:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:44:33 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:44:33 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: dial tcp ->10.242.139.247:8300: getsockopt: connection refused 2021/04/08 15:44:33 [INFO] raft: aborting pipeline replication to peer {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300} 2021/04/08 15:44:33 [INFO] raft: pipelining replication to peer {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300} 2021/04/08 15:44:34 [ERR] memberlist: Failed fallback ping: write tcp 10.242.254.59:48290->10.242.139.200:8302: i/o timeout 2021/04/08 15:44:34 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:44:34 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:44:34 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: dial tcp ->10.242.139.247:8300: getsockopt: connection refused 2021/04/08 15:44:34 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: read tcp 10.242.254.59:47576->10.242.139.200:8300: i/o timeout 2021/04/08 15:44:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:44:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:44:43 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:44:43 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: dial tcp ->10.242.139.247:8300: getsockopt: connection refused 2021/04/08 15:44:45 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:44:45 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: dial tcp ->10.242.139.247:8300: getsockopt: connection refused 2021/04/08 15:44:46 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-w5szm 10.242.139.238 2021/04/08 15:44:46 [INFO] consul: member 'dev-consul-7467bf7b46-w5szm' joined, marking health alive 2021/04/08 15:44:50 [INFO] autopilot: Promoting Server (ID: "e9afffdd-6cc9-d1e4-c968-ec678a43b132" Address: "10.242.139.200:8300") to voter 2021/04/08 15:44:50 [INFO] raft: Updating configuration with AddStaging (e9afffdd-6cc9-d1e4-c968-ec678a43b132, 10.242.139.200:8300) to [{Suffrage:Voter ID:0e6d1abb-47a7-4121-d0d1-64118dab1223 Address:10.242.12.17:8300} {Suffrage:Voter ID:c52c5261-fd75-2dd5-3bad-46a881682908 Address:10.242.139.247:8300} {Suffrage:Voter ID:3454d0fb-1017-7917-b868-ce15c7ffda2e Address:10.242.254.59:8300} {Suffrage:Voter ID:e9afffdd-6cc9-d1e4-c968-ec678a43b132 Address:10.242.139.200:8300}] 2021/04/08 15:44:50 [INFO] autopilot: Attempting removal of stale Server (ID: "c52c5261-fd75-2dd5-3bad-46a881682908" Address: "10.242.139.247:8300") 2021/04/08 15:44:50 [INFO] raft: Updating configuration with RemoveServer (c52c5261-fd75-2dd5-3bad-46a881682908, ) to [{Suffrage:Voter ID:0e6d1abb-47a7-4121-d0d1-64118dab1223 Address:10.242.12.17:8300} {Suffrage:Voter ID:3454d0fb-1017-7917-b868-ce15c7ffda2e Address:10.242.254.59:8300} {Suffrage:Voter ID:e9afffdd-6cc9-d1e4-c968-ec678a43b132 Address:10.242.139.200:8300}] 2021/04/08 15:44:50 [INFO] raft: Removed peer c52c5261-fd75-2dd5-3bad-46a881682908, stopping replication after 1620 2021/04/08 15:44:54 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:44:54 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: dial tcp ->10.242.139.247:8300: getsockopt: connection refused 2021/04/08 15:44:56 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:44:56 [ERR] raft: Failed to heartbeat to 10.242.139.247:8300: dial tcp ->10.242.139.247:8300: getsockopt: connection refused 2021/04/08 15:45:04 [WARN] Unable to get address for server id c52c5261-fd75-2dd5-3bad-46a881682908, using fallback address 10.242.139.247:8300: Could not find address for server id c52c5261-fd75-2dd5-3bad-46a881682908 2021/04/08 15:45:04 [ERR] raft: Failed to AppendEntries to {Voter c52c5261-fd75-2dd5-3bad-46a881682908 10.242.139.247:8300}: dial tcp ->10.242.139.247:8300: getsockopt: connection refused 2021/04/08 15:45:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48398->10.242.139.238:8301: i/o timeout 2021/04/08 15:45:52 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:45:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48416->10.242.139.238:8301: i/o timeout 2021/04/08 15:45:55 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:45:57 [INFO] memberlist: Marking dev-consul-7467bf7b46-w5szm as failed, suspect timeout reached (2 peer confirmations) 2021/04/08 15:45:57 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-w5szm 10.242.139.238 2021/04/08 15:45:57 [INFO] consul: member 'dev-consul-7467bf7b46-w5szm' failed, marking health critical 2021/04/08 15:45:59 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-w5szm 10.242.139.238 2021/04/08 15:45:59 [INFO] consul: member 'dev-consul-7467bf7b46-w5szm' joined, marking health alive 2021/04/08 15:46:01 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:46:01 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 15:46:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48464->10.242.139.238:8301: i/o timeout 2021/04/08 15:46:06 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:46:07 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:46:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48498->10.242.139.238:8301: i/o timeout 2021/04/08 15:46:13 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:46:15 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:46:19 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:46:19 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 15:46:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48538->10.242.139.238:8301: i/o timeout 2021/04/08 15:46:21 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:46:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:51502->10.242.139.200:8301: i/o timeout 2021/04/08 15:46:28 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:50:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:52452->10.242.139.200:8301: i/o timeout 2021/04/08 15:50:16 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:50:17 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:50:17 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 15:50:17 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 15:50:18 [WARN] raft: Failed to contact e9afffdd-6cc9-d1e4-c968-ec678a43b132 in 2.500206489s 2021/04/08 15:50:18 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-w5szm) 2021/04/08 15:50:21 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:50:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:52484->10.242.139.200:8301: i/o timeout 2021/04/08 15:50:21 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:50:23 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:50:23 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 15:50:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:50:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49582->10.242.139.238:8301: i/o timeout 2021/04/08 15:50:25 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:50:29 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:50:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49602->10.242.139.238:8301: i/o timeout 2021/04/08 15:50:29 [INFO] memberlist: Suspect dev-consul-7467bf7b46-w5szm has failed, no acks received 2021/04/08 15:50:29 [INFO] memberlist: Marking dev-consul-7467bf7b46-w5szm as failed, suspect timeout reached (2 peer confirmations) 2021/04/08 15:50:29 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-w5szm 10.242.139.238 2021/04/08 15:50:29 [INFO] consul: member 'dev-consul-7467bf7b46-w5szm' failed, marking health critical 2021/04/08 15:50:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:50:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:50:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:52536->10.242.139.200:8301: i/o timeout 2021/04/08 15:50:32 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:50:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:52552->10.242.139.200:8301: i/o timeout 2021/04/08 15:50:34 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:50:35 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:50:35 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 15:50:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:50:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:52556->10.242.139.200:8301: i/o timeout 2021/04/08 15:50:36 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:50:38 [WARN] raft: Failed to contact e9afffdd-6cc9-d1e4-c968-ec678a43b132 in 2.500239505s 2021/04/08 15:50:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:50:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:52578->10.242.139.200:8301: i/o timeout 2021/04/08 15:50:39 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:50:40 [WARN] raft: Failed to contact e9afffdd-6cc9-d1e4-c968-ec678a43b132 in 4.956799494s 2021/04/08 15:50:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:50:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:52586->10.242.139.200:8301: i/o timeout 2021/04/08 15:50:41 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received 2021/04/08 15:50:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:50:42 [WARN] raft: Failed to contact e9afffdd-6cc9-d1e4-c968-ec678a43b132 in 7.397760424s 2021/04/08 15:50:44 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations) 2021/04/08 15:50:44 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.139.200 2021/04/08 15:50:44 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.139.200:8300) (DC: dc1) 2021/04/08 15:50:44 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical 2021/04/08 15:50:44 [ERR] raft: Failed to pipeline AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: write tcp 10.242.254.59:47656->10.242.139.200:8300: use of closed network connection 2021/04/08 15:50:44 [INFO] raft: aborting pipeline replication to peer {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300} 2021/04/08 15:50:44 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:50:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:50:46 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: read tcp 10.242.254.59:47668->10.242.139.200:8300: i/o timeout 2021/04/08 15:50:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:50:46 [WARN] raft: Failed to contact e9afffdd-6cc9-d1e4-c968-ec678a43b132 in 2.500471386s 2021/04/08 15:50:47 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:50:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:50:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:49980->10.242.139.200:8302: i/o timeout 2021/04/08 15:50:48 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:50:49 [WARN] raft: Failed to contact e9afffdd-6cc9-d1e4-c968-ec678a43b132 in 4.937189298s 2021/04/08 15:50:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:50:51 [WARN] raft: Failed to contact e9afffdd-6cc9-d1e4-c968-ec678a43b132 in 7.435573664s 2021/04/08 15:50:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:50:53 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.139.200:8301 2021/04/08 15:50:54 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: read tcp 10.242.254.59:49304->10.242.139.200:8300: i/o timeout 2021/04/08 15:50:54 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:50:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:50:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:50:57 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: read tcp 10.242.254.59:49316->10.242.139.200:8300: i/o timeout 2021/04/08 15:50:58 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:50:58 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 15:50:58 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.139.200:36906 2021/04/08 15:50:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:00 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 15:51:00 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF 2021/04/08 15:51:01 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:51:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:50050->10.242.139.200:8302: i/o timeout 2021/04/08 15:51:03 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:51:04 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: read tcp 10.242.254.59:49350->10.242.139.200:8300: i/o timeout 2021/04/08 15:51:04 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:51:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:08 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: read tcp 10.242.254.59:49366->10.242.139.200:8300: i/o timeout 2021/04/08 15:51:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:08 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:51:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:12 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/04/08 15:51:12 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.139.200 2021/04/08 15:51:12 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan" 2021/04/08 15:51:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:50076->10.242.139.200:8302: i/o timeout 2021/04/08 15:51:13 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received 2021/04/08 15:51:14 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: read tcp 10.242.254.59:49400->10.242.139.200:8300: i/o timeout 2021/04/08 15:51:14 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:51:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:18 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: read tcp 10.242.254.59:49422->10.242.139.200:8300: i/o timeout 2021/04/08 15:51:19 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:51:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:24 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: read tcp 10.242.254.59:49448->10.242.139.200:8300: i/o timeout 2021/04/08 15:51:24 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:51:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:29 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: read tcp 10.242.254.59:49470->10.242.139.200:8300: i/o timeout 2021/04/08 15:51:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:30 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:51:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:33 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-w5szm 10.242.139.238:8301 2021/04/08 15:51:34 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: read tcp 10.242.254.59:49496->10.242.139.200:8300: i/o timeout 2021/04/08 15:51:34 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:51:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:40 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 15:51:40 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF 2021/04/08 15:51:40 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: read tcp 10.242.254.59:49520->10.242.139.200:8300: i/o timeout 2021/04/08 15:51:41 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:51:43 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:51:44 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: read tcp 10.242.254.59:49540->10.242.139.200:8300: i/o timeout 2021/04/08 15:51:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:44 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:51:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:51 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: read tcp 10.242.254.59:49572->10.242.139.200:8300: i/o timeout 2021/04/08 15:51:52 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:51:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:54 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: read tcp 10.242.254.59:49592->10.242.139.200:8300: i/o timeout 2021/04/08 15:51:55 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:51:55 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.139.200:8302 2021/04/08 15:51:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:51:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:02 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: read tcp 10.242.254.59:49626->10.242.139.200:8300: i/o timeout 2021/04/08 15:52:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:03 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:52:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:05 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: read tcp 10.242.254.59:49638->10.242.139.200:8300: i/o timeout 2021/04/08 15:52:05 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:52:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:13 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.139.200:8301 2021/04/08 15:52:13 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: read tcp 10.242.254.59:49678->10.242.139.200:8300: i/o timeout 2021/04/08 15:52:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:15 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: read tcp 10.242.254.59:49688->10.242.139.200:8300: i/o timeout 2021/04/08 15:52:16 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:52:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:17 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:52:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:22 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 15:52:22 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF 2021/04/08 15:52:25 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:52:26 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: read tcp 10.242.254.59:49734->10.242.139.200:8300: i/o timeout 2021/04/08 15:52:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:27 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: read tcp 10.242.254.59:49742->10.242.139.200:8300: i/o timeout 2021/04/08 15:52:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:29 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:52:29 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:52:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:39 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: read tcp 10.242.254.59:49800->10.242.139.200:8300: i/o timeout 2021/04/08 15:52:39 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: read tcp 10.242.254.59:49802->10.242.139.200:8300: i/o timeout 2021/04/08 15:52:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:44 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:52:45 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:52:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:53 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.139.200:8301 2021/04/08 15:52:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:54 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: read tcp 10.242.254.59:49870->10.242.139.200:8300: i/o timeout 2021/04/08 15:52:55 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: read tcp 10.242.254.59:49874->10.242.139.200:8300: i/o timeout 2021/04/08 15:52:55 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.139.200:8302 2021/04/08 15:52:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:52:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:04 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 15:53:04 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF 2021/04/08 15:53:05 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:53:06 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:53:07 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:53:07 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 15:53:07 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 15:53:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:15 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: read tcp 10.242.254.59:49960->10.242.139.200:8300: i/o timeout 2021/04/08 15:53:16 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: read tcp 10.242.254.59:49962->10.242.139.200:8300: i/o timeout 2021/04/08 15:53:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:25 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:53:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:27 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:53:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:33 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-w5szm 10.242.139.238:8301 2021/04/08 15:53:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:35 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: read tcp 10.242.254.59:50052->10.242.139.200:8300: i/o timeout 2021/04/08 15:53:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:37 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: read tcp 10.242.254.59:50064->10.242.139.200:8300: i/o timeout 2021/04/08 15:53:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:45 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:53:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:46 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 15:53:46 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF 2021/04/08 15:53:48 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:53:49 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:53:49 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 15:53:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:55 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: read tcp 10.242.254.59:50142->10.242.139.200:8300: i/o timeout 2021/04/08 15:53:55 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.139.200:8302 2021/04/08 15:53:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:53:58 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: read tcp 10.242.254.59:50158->10.242.139.200:8300: i/o timeout 2021/04/08 15:53:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:05 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:54:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:09 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:54:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:13 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-w5szm 10.242.139.238:8301 2021/04/08 15:54:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:15 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: read tcp 10.242.254.59:50230->10.242.139.200:8300: i/o timeout 2021/04/08 15:54:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:19 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: read tcp 10.242.254.59:50250->10.242.139.200:8300: i/o timeout 2021/04/08 15:54:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:26 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:54:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:28 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 15:54:28 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF 2021/04/08 15:54:30 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:54:31 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:54:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:36 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: read tcp 10.242.254.59:50322->10.242.139.200:8300: i/o timeout 2021/04/08 15:54:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:40 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: read tcp 10.242.254.59:50344->10.242.139.200:8300: i/o timeout 2021/04/08 15:54:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:46 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:54:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:51 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:54:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:53 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-w5szm 10.242.139.238:8301 2021/04/08 15:54:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:56 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: read tcp 10.242.254.59:50414->10.242.139.200:8300: i/o timeout 2021/04/08 15:54:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:54:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:01 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: read tcp 10.242.254.59:50436->10.242.139.200:8300: i/o timeout 2021/04/08 15:55:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:06 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:55:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:10 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 15:55:10 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF 2021/04/08 15:55:12 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:55:13 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:55:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:16 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: read tcp 10.242.254.59:50500->10.242.139.200:8300: i/o timeout 2021/04/08 15:55:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:22 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: read tcp 10.242.254.59:50534->10.242.139.200:8300: i/o timeout 2021/04/08 15:55:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:26 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:55:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:33 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:55:33 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-w5szm 10.242.139.238:8301 2021/04/08 15:55:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:36 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: read tcp 10.242.254.59:50594->10.242.139.200:8300: i/o timeout 2021/04/08 15:55:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:43 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: read tcp 10.242.254.59:50622->10.242.139.200:8300: i/o timeout 2021/04/08 15:55:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:47 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:55:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:52 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 15:55:52 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF 2021/04/08 15:55:54 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:55:55 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:55:55 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.139.200:8302 2021/04/08 15:55:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:55:57 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: read tcp 10.242.254.59:50686->10.242.139.200:8300: i/o timeout 2021/04/08 15:55:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:04 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: read tcp 10.242.254.59:50720->10.242.139.200:8300: i/o timeout 2021/04/08 15:56:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:07 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:56:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:13 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.139.200:8301 2021/04/08 15:56:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:15 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:56:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:17 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: read tcp 10.242.254.59:50778->10.242.139.200:8300: i/o timeout 2021/04/08 15:56:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:25 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: read tcp 10.242.254.59:50822->10.242.139.200:8300: i/o timeout 2021/04/08 15:56:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:27 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:56:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:34 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/08 15:56:34 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF 2021/04/08 15:56:36 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:56:37 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: read tcp 10.242.254.59:50996->10.242.139.200:8300: i/o timeout 2021/04/08 15:56:37 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:56:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:46 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:56:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:46 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:56:47 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:56:49 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:56:49 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 15:56:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:53 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-w5szm 10.242.139.238:8301 2021/04/08 15:56:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:55 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.139.200:8302 2021/04/08 15:56:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:57 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:56:57 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:56:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:56:58 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:56:59 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-fs8vl 10.242.254.50 2021/04/08 15:56:59 [INFO] consul: member 'dev-consul-7467bf7b46-fs8vl' joined, marking health alive 2021/04/08 15:57:01 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:57:01 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 15:57:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:07 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:57:08 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:57:09 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:10 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:57:13 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:57:13 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 15:57:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:18 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:57:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:19 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:57:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:22 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:57:25 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:57:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:28 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:57:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:29 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:57:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:34 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:57:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:37 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:57:37 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 15:57:38 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:57:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:40 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:57:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:46 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:57:48 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:57:49 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:57:50 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:57:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:57:58 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:57:58 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:57:59 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:58:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:01 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:58:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:03 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.139.200:8301 2021/04/08 15:58:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:08 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:58:08 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:58:09 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:58:09 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/04/08 15:58:11 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:11 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:58:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:18 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:58:18 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:58:22 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:58:22 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:58:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48420->10.242.254.50:8301: i/o timeout 2021/04/08 15:58:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:29 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:29 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:58:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:31 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:58:32 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:58:33 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:58:33 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/04/08 15:58:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48702->10.242.254.50:8301: i/o timeout 2021/04/08 15:58:36 [INFO] memberlist: Suspect dev-consul-7467bf7b46-fs8vl has failed, no acks received 2021/04/08 15:58:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.254.59:48742->10.242.254.50:8301: i/o timeout 2021/04/08 15:58:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:39 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:58:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:42 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:58:43 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:58:43 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.139.200:8301 2021/04/08 15:58:46 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:58:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:50 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:58:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:53 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:58:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:58:55 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:58:57 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:58:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:59:00 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:59:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:59:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:59:04 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:59:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:59:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:59:06 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:59:09 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:59:10 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:59:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:59:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:59:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:59:14 [ERR] raft: Failed to heartbeat to 10.242.139.200:8300: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:59:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:59:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:59:18 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:59:20 [ERR] raft: Failed to AppendEntries to {Nonvoter e9afffdd-6cc9-d1e4-c968-ec678a43b132 10.242.139.200:8300}: dial tcp ->10.242.139.200:8300: i/o timeout 2021/04/08 15:59:21 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded 2021/04/08 15:59:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:59:23 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-w5szm 10.242.139.238:8301 2021/04/08 15:59:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding 2021/04/08 15:59:25 [WARN] Unable to get address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132, using fallback address 10.242.139.200:8300: Could not find address for server id e9afffdd-6cc9-d1e4-c968-ec678a43b132 2021/04/08 15:59:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding