By type
+ CONSUL_BIND=
+ [ -n ]
+ CONSUL_CLIENT=
+ [ -n ]
+ CONSUL_DATA_DIR=/consul/data
+ CONSUL_CONFIG_DIR=/consul/config
+ [ -n ]
+ [ a = - ]
+ [ agent = agent ]
+ shift
+ set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
+ [ consul = consul ]
+ stat -c %u /consul/data
+ id -u consul
+ [ 100 != 100 ]
+ stat -c %u /consul/config
+ id -u consul
+ [ 100 != 100 ]
+ [ ! -z ]
+ exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
bootstrap_expect > 0: expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
Version: 'v1.0.6'
Node ID: 'b3d9ccb9-04c2-7b7e-2bea-552e392b6c00'
Node name: 'dev-consul-server-1'
Datacenter: 'dc1' (Segment: '')
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.242.210.71 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2021/05/09 22:40:06 [INFO] raft: Initial configuration (index=0): []
2021/05/09 22:40:06 [INFO] raft: Node at 10.242.210.71:8300 [Follower] entering Follower state (Leader: "")
2021/05/09 22:40:06 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.210.71
2021/05/09 22:40:06 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.210.71
2021/05/09 22:40:06 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/09 22:40:06 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.210.71:8300) (DC: dc1)
2021/05/09 22:40:06 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2021/05/09 22:40:06 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2021/05/09 22:40:06 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
2021/05/09 22:40:06 [INFO] agent: started state syncer
2021/05/09 22:40:06 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer
2021/05/09 22:40:06 [INFO] agent: Joining LAN cluster...
2021/05/09 22:40:06 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/05/09 22:40:06 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.61.70
2021/05/09 22:40:06 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.61.70:8300) (DC: dc1)
2021/05/09 22:40:06 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.61.70
2021/05/09 22:40:06 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/09 22:40:06 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
2021/05/09 22:40:06 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/05/09 22:40:06 [INFO] agent: (LAN) joined: 1 Err:
2021/05/09 22:40:06 [INFO] agent: Join LAN completed. Synced with 1 initial agents
==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=3ae416c3-9d3c-7e2f-0133-56f3d85f3d9e&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2021/05/09 22:40:13 [ERR] agent: failed to sync remote state: No cluster leader
2021/05/09 22:40:13 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 22:40:13 [WARN] raft: no known peers, aborting election
2021/05/09 22:40:24 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/09 22:40:24 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 22:40:24 [INFO] consul: Existing Raft peers reported by dev-consul-server-2, disabling bootstrap mode
2021/05/09 22:40:24 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.163.157
2021/05/09 22:40:24 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/05/09 22:40:25 [DEBUG] raft-net: 10.242.210.71:8300 accepted connection from: 10.242.61.70:46208
2021/05/09 22:40:25 [WARN] raft: Failed to get previous log: 1 log not found (last: 0)
2021/05/09 22:40:25 [INFO] consul: New leader elected: dev-consul-server-0
2021/05/09 22:40:26 [DEBUG] raft-net: 10.242.210.71:8300 accepted connection from: 10.242.61.70:46214
2021/05/09 22:40:27 [INFO] agent: Synced node info
2021/05/09 23:25:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:36444->10.242.163.157:8301: i/o timeout
2021/05/09 23:25:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:52614->10.242.163.138:8301: i/o timeout
2021/05/09 23:25:15 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:25:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:36494->10.242.163.157:8301: i/o timeout
2021/05/09 23:25:17 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:25:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:52662->10.242.163.138:8301: i/o timeout
2021/05/09 23:25:19 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:25:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:36576->10.242.163.157:8301: i/o timeout
2021/05/09 23:25:21 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:25:22 [ERR] memberlist: Push/Pull with dev-consul-7467bf7b46-xplxm failed: read tcp 10.242.210.71:52574->10.242.163.138:8301: i/o timeout
2021/05/09 23:25:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:52726->10.242.163.138:8301: i/o timeout
2021/05/09 23:25:23 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:25:24 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/09 23:25:24 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:25:24 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/09 23:25:24 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:25:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:52776->10.242.163.138:8301: i/o timeout
2021/05/09 23:25:26 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:25:26 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (1 peer confirmations)
2021/05/09 23:25:26 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:25:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:36666->10.242.163.157:8301: i/o timeout
2021/05/09 23:25:29 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:25:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:36718->10.242.163.157:8301: i/o timeout
2021/05/09 23:25:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:36744->10.242.163.157:8301: i/o timeout
2021/05/09 23:25:34 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:25:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:36758->10.242.163.157:8301: i/o timeout
2021/05/09 23:25:36 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:25:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:36820->10.242.163.157:8301: i/o timeout
2021/05/09 23:25:39 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:25:40 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:25:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:60268->10.242.163.157:8302: i/o timeout
2021/05/09 23:25:41 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/09 23:25:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:53010->10.242.163.138:8301: i/o timeout
2021/05/09 23:25:42 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:25:45 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/09 23:25:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:53110->10.242.163.138:8301: i/o timeout
2021/05/09 23:25:49 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:25:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37002->10.242.163.157:8301: i/o timeout
2021/05/09 23:25:52 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:25:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37038->10.242.163.157:8301: i/o timeout
2021/05/09 23:25:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:25:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:53212->10.242.163.138:8301: i/o timeout
2021/05/09 23:25:59 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:26:00 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (1 peer confirmations)
2021/05/09 23:26:00 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:26:01 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/05/09 23:26:01 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/05/09 23:26:01 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/09 23:26:01 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:26:01 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/09 23:26:01 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:26:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37158->10.242.163.157:8301: i/o timeout
2021/05/09 23:26:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37182->10.242.163.157:8301: i/o timeout
2021/05/09 23:26:04 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:26:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37210->10.242.163.157:8301: i/o timeout
2021/05/09 23:26:06 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:26:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37274->10.242.163.157:8301: i/o timeout
2021/05/09 23:26:10 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:26:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37298->10.242.163.157:8301: i/o timeout
2021/05/09 23:26:12 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:26:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37328->10.242.163.157:8301: i/o timeout
2021/05/09 23:26:15 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:26:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37384->10.242.163.157:8301: i/o timeout
2021/05/09 23:26:18 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:26:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37402->10.242.163.157:8301: i/o timeout
2021/05/09 23:26:20 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:26:21 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:26:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:53596->10.242.163.138:8301: i/o timeout
2021/05/09 23:26:23 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:26:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:60916->10.242.163.157:8302: i/o timeout
2021/05/09 23:26:26 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:26:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37468->10.242.163.157:8301: i/o timeout
2021/05/09 23:26:26 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:26:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:53652->10.242.163.138:8301: i/o timeout
2021/05/09 23:26:29 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:26:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37576->10.242.163.157:8301: i/o timeout
2021/05/09 23:26:32 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:26:34 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (1 peer confirmations)
2021/05/09 23:26:34 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:26:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:53764->10.242.163.138:8301: i/o timeout
2021/05/09 23:26:35 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:26:35 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/09 23:26:35 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:26:36 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-xplxm 10.242.163.138:8301
2021/05/09 23:26:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37642->10.242.163.157:8301: i/o timeout
2021/05/09 23:26:37 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:26:37 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/09 23:26:37 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.163.138:44678
2021/05/09 23:26:46 [ERR] memberlist: failed to receive: read tcp 10.242.210.71:8301->10.242.163.157:33568: i/o timeout from=10.242.163.157:33568
2021/05/09 23:26:46 [ERR] memberlist: Failed to send error: write tcp 10.242.210.71:8301->10.242.163.157:33568: i/o timeout from=10.242.163.157:33568
2021/05/09 23:26:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:33058->10.242.163.157:8302: i/o timeout
2021/05/09 23:26:51 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:26:56 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/09 23:26:56 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.163.157
2021/05/09 23:26:56 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/05/09 23:27:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:33136->10.242.163.157:8302: i/o timeout
2021/05/09 23:27:01 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:27:04 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.163.157
2021/05/09 23:27:04 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/05/09 23:27:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:33430->10.242.163.157:8302: i/o timeout
2021/05/09 23:27:16 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:27:16 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-xplxm 10.242.163.138:8301
2021/05/09 23:27:25 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:27:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:33598->10.242.163.157:8302: i/o timeout
2021/05/09 23:27:26 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:27:26 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/05/09 23:27:26 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:27:26 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/09 23:27:26 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:27:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:38478->10.242.163.157:8301: i/o timeout
2021/05/09 23:27:29 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:27:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:38516->10.242.163.157:8301: i/o timeout
2021/05/09 23:27:31 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:27:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:38536->10.242.163.157:8301: i/o timeout
2021/05/09 23:27:33 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:27:33 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:28:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:39046->10.242.163.157:8301: i/o timeout
2021/05/09 23:30:28 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-xplxm)
2021/05/09 23:32:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:58440->10.242.163.138:8301: i/o timeout
2021/05/09 23:32:13 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:32:15 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-xplxm)
2021/05/09 23:34:10 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-xplxm but other probes failed, network may be misconfigured
2021/05/09 23:34:21 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-xplxm)
2021/05/09 23:35:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:60714->10.242.163.138:8301: i/o timeout
2021/05/09 23:35:03 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:35:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:33114->10.242.163.138:8301: i/o timeout
2021/05/09 23:35:49 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:35:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:33246->10.242.163.138:8301: i/o timeout
2021/05/09 23:35:59 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:36:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:45364->10.242.163.157:8301: i/o timeout
2021/05/09 23:36:02 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:36:03 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (2 peer confirmations)
2021/05/09 23:36:03 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:36:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:45494->10.242.163.157:8301: i/o timeout
2021/05/09 23:36:10 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:36:11 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-xplxm)
2021/05/09 23:36:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:45506->10.242.163.157:8301: i/o timeout
2021/05/09 23:36:12 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:36:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:45546->10.242.163.157:8301: i/o timeout
2021/05/09 23:36:15 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:36:15 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:36:15 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/05/09 23:36:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:45714->10.242.163.157:8301: i/o timeout
2021/05/09 23:36:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:45730->10.242.163.157:8301: i/o timeout
2021/05/09 23:36:28 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:39:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:36302->10.242.163.138:8301: i/o timeout
2021/05/09 23:39:48 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:40:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37082->10.242.163.138:8301: i/o timeout
2021/05/09 23:40:47 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:41:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37538->10.242.163.138:8301: i/o timeout
2021/05/09 23:41:21 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:41:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37548->10.242.163.138:8301: i/o timeout
2021/05/09 23:41:24 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:41:24 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:41:25 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:41:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37662->10.242.163.138:8301: i/o timeout
2021/05/09 23:41:30 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:41:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37698->10.242.163.138:8301: i/o timeout
2021/05/09 23:41:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:41:34 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (2 peer confirmations)
2021/05/09 23:41:34 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:41:35 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:41:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37774->10.242.163.138:8301: i/o timeout
2021/05/09 23:41:39 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:41:41 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:41:50 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-xplxm)
2021/05/09 23:41:50 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:41:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:50102->10.242.163.157:8301: i/o timeout
2021/05/09 23:41:56 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:43:22 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/05/09 23:44:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:40332->10.242.163.138:8301: i/o timeout
2021/05/09 23:44:51 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:44:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:40414->10.242.163.138:8301: i/o timeout
2021/05/09 23:44:58 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:45:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:52576->10.242.163.157:8301: i/o timeout
2021/05/09 23:45:02 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:45:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:40510->10.242.163.138:8301: i/o timeout
2021/05/09 23:45:04 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:45:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:40614->10.242.163.138:8301: i/o timeout
2021/05/09 23:45:12 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:45:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:40630->10.242.163.138:8301: i/o timeout
2021/05/09 23:45:15 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:45:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:52766->10.242.163.157:8301: i/o timeout
2021/05/09 23:45:18 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:45:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:40730->10.242.163.138:8301: i/o timeout
2021/05/09 23:45:20 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:45:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:52838->10.242.163.157:8301: i/o timeout
2021/05/09 23:45:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:45:22 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (1 peer confirmations)
2021/05/09 23:45:22 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:45:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:40762->10.242.163.138:8301: i/o timeout
2021/05/09 23:45:24 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:45:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:48078->10.242.163.157:8302: i/o timeout
2021/05/09 23:45:26 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:45:26 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-xplxm 10.242.163.138:8301
2021/05/09 23:45:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:52908->10.242.163.157:8301: i/o timeout
2021/05/09 23:45:27 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:45:28 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/09 23:45:28 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:45:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:52938->10.242.163.157:8301: i/o timeout
2021/05/09 23:45:29 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:45:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:48156->10.242.163.157:8302: i/o timeout
2021/05/09 23:45:36 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:45:39 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/09 23:45:48 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/09 23:45:48 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.163.138:43262
2021/05/09 23:45:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:48440->10.242.163.157:8302: i/o timeout
2021/05/09 23:45:51 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:45:53 [ERR] memberlist: failed to receive: read tcp 10.242.210.71:8301->10.242.163.157:57164: i/o timeout from=10.242.163.157:57164
2021/05/09 23:45:53 [ERR] memberlist: Failed to send error: write tcp 10.242.210.71:8301->10.242.163.157:57164: i/o timeout from=10.242.163.157:57164
2021/05/09 23:45:53 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/09 23:45:53 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.163.157
2021/05/09 23:45:53 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/05/09 23:45:55 [ERR] memberlist: failed to receive: read tcp 10.242.210.71:8301->10.242.163.138:34654: i/o timeout from=10.242.163.138:34654
2021/05/09 23:45:55 [ERR] memberlist: Failed to send error: write tcp 10.242.210.71:8301->10.242.163.138:34654: i/o timeout from=10.242.163.138:34654
2021/05/09 23:45:57 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:45:59 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/09 23:45:59 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:45:59 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.163.157
2021/05/09 23:45:59 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/05/09 23:46:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:41298->10.242.163.138:8301: i/o timeout
2021/05/09 23:46:00 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:46:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:41322->10.242.163.138:8301: i/o timeout
2021/05/09 23:46:03 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:46:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:53446->10.242.163.157:8301: i/o timeout
2021/05/09 23:46:06 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:46:07 [DEBUG] raft-net: 10.242.210.71:8300 accepted connection from: 10.242.163.157:56826
2021/05/09 23:46:08 [WARN] raft: Rejecting vote request from 10.242.163.157:8300 since we have a leader: 10.242.61.70:8300
2021/05/09 23:46:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:53492->10.242.163.157:8301: i/o timeout
2021/05/09 23:46:08 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:46:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:41418->10.242.163.138:8301: i/o timeout
2021/05/09 23:46:10 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:46:11 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (1 peer confirmations)
2021/05/09 23:46:11 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:46:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:41458->10.242.163.138:8301: i/o timeout
2021/05/09 23:46:12 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:46:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:53568->10.242.163.157:8301: i/o timeout
2021/05/09 23:46:14 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:46:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:48798->10.242.163.157:8302: i/o timeout
2021/05/09 23:46:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:53614->10.242.163.157:8301: i/o timeout
2021/05/09 23:46:20 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:46:32 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/09 23:46:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:41758->10.242.163.138:8301: i/o timeout
2021/05/09 23:46:33 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:46:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:41772->10.242.163.138:8301: i/o timeout
2021/05/09 23:46:36 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:46:36 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-xplxm)
2021/05/09 23:47:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:42862->10.242.163.138:8301: i/o timeout
2021/05/09 23:47:55 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:47:57 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (2 peer confirmations)
2021/05/09 23:47:57 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:47:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:54974->10.242.163.157:8301: i/o timeout
2021/05/09 23:47:58 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:48:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:55040->10.242.163.157:8301: i/o timeout
2021/05/09 23:48:01 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:48:03 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/09 23:48:04 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-xplxm)
2021/05/09 23:48:04 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:48:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:43008->10.242.163.138:8301: i/o timeout
2021/05/09 23:48:07 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:48:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:55134->10.242.163.157:8301: i/o timeout
2021/05/09 23:48:10 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:48:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:55192->10.242.163.157:8301: i/o timeout
2021/05/09 23:48:13 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:48:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:50430->10.242.163.157:8302: i/o timeout
2021/05/09 23:48:16 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:48:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:43122->10.242.163.138:8301: i/o timeout
2021/05/09 23:48:16 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:48:18 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (1 peer confirmations)
2021/05/09 23:48:18 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:48:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:43182->10.242.163.138:8301: i/o timeout
2021/05/09 23:48:19 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:48:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:55316->10.242.163.157:8301: i/o timeout
2021/05/09 23:48:21 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:48:21 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/05/09 23:48:21 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/09 23:48:21 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:48:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:50572->10.242.163.157:8302: i/o timeout
2021/05/09 23:48:26 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:48:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:50650->10.242.163.157:8302: i/o timeout
2021/05/09 23:48:36 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:48:36 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.163.157:8301
2021/05/09 23:48:49 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:48:50 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/09 23:48:50 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:48:50 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/05/09 23:48:50 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:48:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:43668->10.242.163.138:8301: i/o timeout
2021/05/09 23:48:51 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:48:52 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:49:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:43814->10.242.163.138:8301: i/o timeout
2021/05/09 23:49:00 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:49:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:55948->10.242.163.157:8301: i/o timeout
2021/05/09 23:49:04 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:49:05 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/05/09 23:49:06 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-xplxm but other probes failed, network may be misconfigured
2021/05/09 23:49:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:56010->10.242.163.157:8301: i/o timeout
2021/05/09 23:49:08 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:49:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:43944->10.242.163.138:8301: i/o timeout
2021/05/09 23:49:10 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:49:10 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (2 peer confirmations)
2021/05/09 23:49:10 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:49:13 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:49:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:56106->10.242.163.157:8301: i/o timeout
2021/05/09 23:49:14 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:49:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:44028->10.242.163.138:8301: i/o timeout
2021/05/09 23:49:16 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:49:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:56150->10.242.163.157:8301: i/o timeout
2021/05/09 23:49:18 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:49:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:44132->10.242.163.138:8301: i/o timeout
2021/05/09 23:49:21 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:49:23 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/09 23:49:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:44156->10.242.163.138:8301: i/o timeout
2021/05/09 23:49:24 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:49:25 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/05/09 23:49:25 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/09 23:49:25 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:49:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:56270->10.242.163.157:8301: i/o timeout
2021/05/09 23:49:26 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:49:27 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:49:27 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/09 23:49:27 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:49:29 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:50:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:45252->10.242.163.138:8301: i/o timeout
2021/05/09 23:50:45 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:56:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:50060->10.242.163.138:8301: i/o timeout
2021/05/09 23:56:47 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:56:50 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-xplxm)
2021/05/09 23:56:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:50148->10.242.163.138:8301: i/o timeout
2021/05/09 23:56:53 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:56:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:50174->10.242.163.138:8301: i/o timeout
2021/05/09 23:56:59 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/09 23:57:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:50360->10.242.163.138:8301: i/o timeout
2021/05/09 23:57:09 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:57:14 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-xplxm)
2021/05/09 23:57:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:50932->10.242.163.138:8301: i/o timeout
2021/05/09 23:57:51 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:58:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:51098->10.242.163.138:8301: i/o timeout
2021/05/09 23:58:05 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:58:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:51218->10.242.163.138:8301: i/o timeout
2021/05/09 23:58:14 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:58:17 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (2 peer confirmations)
2021/05/09 23:58:17 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:58:18 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-xplxm but other probes failed, network may be misconfigured
2021/05/09 23:58:18 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:58:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:51324->10.242.163.138:8301: i/o timeout
2021/05/09 23:58:20 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:58:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:35208->10.242.163.157:8301: i/o timeout
2021/05/09 23:58:23 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:58:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:51360->10.242.163.138:8301: i/o timeout
2021/05/09 23:58:25 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:58:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:35276->10.242.163.157:8301: i/o timeout
2021/05/09 23:58:28 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:58:28 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:58:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:35330->10.242.163.157:8301: i/o timeout
2021/05/09 23:58:32 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:58:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:35358->10.242.163.157:8301: i/o timeout
2021/05/09 23:58:34 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:58:37 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-xplxm but other probes failed, network may be misconfigured
2021/05/09 23:58:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:51552->10.242.163.138:8301: i/o timeout
2021/05/09 23:58:39 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:58:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:35466->10.242.163.157:8301: i/o timeout
2021/05/09 23:58:42 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:58:44 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/09 23:58:45 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (2 peer confirmations)
2021/05/09 23:58:45 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:58:46 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/05/09 23:58:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:51664->10.242.163.138:8301: i/o timeout
2021/05/09 23:58:46 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:58:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:35558->10.242.163.157:8301: i/o timeout
2021/05/09 23:58:49 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:58:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:35612->10.242.163.157:8301: i/o timeout
2021/05/09 23:58:51 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:58:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:35634->10.242.163.157:8301: i/o timeout
2021/05/09 23:58:53 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:58:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:35650->10.242.163.157:8301: i/o timeout
2021/05/09 23:58:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:58:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:35690->10.242.163.157:8301: i/o timeout
2021/05/09 23:58:57 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:58:58 [ERR] memberlist: Push/Pull with dev-consul-server-2 failed: read tcp 10.242.210.71:35574->10.242.163.157:8301: i/o timeout
2021/05/09 23:58:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:35720->10.242.163.157:8301: i/o timeout
2021/05/09 23:58:59 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:00 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/09 23:59:00 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:59:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:59170->10.242.163.157:8302: i/o timeout
2021/05/09 23:59:01 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/09 23:59:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:35756->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:01 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:09 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/09 23:59:09 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:59:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:35914->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:12 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:12 [WARN] raft: Rejecting vote request from 10.242.163.157:8300 since we have a leader: 10.242.61.70:8300
2021/05/09 23:59:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:35930->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:14 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:35990->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:18 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:36034->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:20 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:36062->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:22 [INFO] consul: New leader elected: dev-consul-server-0
2021/05/09 23:59:22 [DEBUG] raft-net: 10.242.210.71:8300 accepted connection from: 10.242.61.70:50628
2021/05/09 23:59:23 [WARN] raft: Rejecting vote request from 10.242.163.157:8300 since we have a leader: 10.242.61.70:8300
2021/05/09 23:59:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:36078->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:24 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:25 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/05/09 23:59:25 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/09 23:59:25 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:59:26 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:59:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:52272->10.242.163.138:8301: i/o timeout
2021/05/09 23:59:28 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:59:28 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/05/09 23:59:28 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/09 23:59:28 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:59:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:36174->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:30 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:52338->10.242.163.138:8301: i/o timeout
2021/05/09 23:59:32 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:59:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:36210->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:34 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:36254->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:36 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:52402->10.242.163.138:8301: i/o timeout
2021/05/09 23:59:38 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:59:39 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (1 peer confirmations)
2021/05/09 23:59:39 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:59:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:36294->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:40 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:59666->10.242.163.157:8302: i/o timeout
2021/05/09 23:59:41 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/05/09 23:59:41 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/09 23:59:41 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:59:45 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/05/09 23:59:46 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/05/09 23:59:46 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.163.157:8301
2021/05/09 23:59:46 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/09 23:59:46 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/09 23:59:47 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/09 23:59:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:36442->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:49 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/09 23:59:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:52608->10.242.163.138:8301: i/o timeout
2021/05/09 23:59:52 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:59:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:52638->10.242.163.138:8301: i/o timeout
2021/05/09 23:59:55 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/09 23:59:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:36552->10.242.163.157:8301: i/o timeout
2021/05/09 23:59:58 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:00:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:36612->10.242.163.157:8301: i/o timeout
2021/05/10 00:00:00 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:00:00 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/05/10 00:00:00 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:00:00 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:00:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:52760->10.242.163.138:8301: i/o timeout
2021/05/10 00:00:02 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:00:03 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (1 peer confirmations)
2021/05/10 00:00:03 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:00:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:52784->10.242.163.138:8301: i/o timeout
2021/05/10 00:00:04 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:00:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:60192->10.242.163.157:8302: i/o timeout
2021/05/10 00:00:11 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:00:13 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/05/10 00:00:13 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:00:13 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-xplxm)
2021/05/10 00:00:14 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:00:14 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:00:16 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/10 00:00:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:36920->10.242.163.157:8301: i/o timeout
2021/05/10 00:00:20 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:00:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:53082->10.242.163.138:8301: i/o timeout
2021/05/10 00:00:23 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:00:23 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/05/10 00:00:23 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:00:23 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:00:25 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/10 00:00:25 [WARN] raft: Rejecting vote request from 10.242.163.157:8300 since we have a leader: 10.242.61.70:8300
2021/05/10 00:00:25 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:00:25 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:00:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:53152->10.242.163.138:8301: i/o timeout
2021/05/10 00:00:28 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:00:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:53196->10.242.163.138:8301: i/o timeout
2021/05/10 00:00:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37070->10.242.163.157:8301: i/o timeout
2021/05/10 00:00:31 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:00:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:53236->10.242.163.138:8301: i/o timeout
2021/05/10 00:00:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:00:34 [WARN] raft: Heartbeat timeout from "10.242.61.70:8300" reached, starting election
2021/05/10 00:00:34 [INFO] raft: Node at 10.242.210.71:8300 [Candidate] entering Candidate state in term 6
2021/05/10 00:00:34 [INFO] raft: Election won. Tally: 2
2021/05/10 00:00:34 [INFO] raft: Node at 10.242.210.71:8300 [Leader] entering Leader state
2021/05/10 00:00:34 [INFO] raft: Added peer 19ce88fc-998f-ee2b-b2d5-6fafec370e52, starting replication
2021/05/10 00:00:34 [INFO] raft: Added peer 6ff779d7-507c-f4ae-1d24-7621a2e1d40c, starting replication
2021/05/10 00:00:34 [INFO] consul: cluster leadership acquired
2021/05/10 00:00:34 [INFO] consul: New leader elected: dev-consul-server-1
2021/05/10 00:00:34 [INFO] raft: pipelining replication to peer {Voter 6ff779d7-507c-f4ae-1d24-7621a2e1d40c 10.242.61.70:8300}
2021/05/10 00:00:34 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/10 00:00:36 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/10 00:00:36 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.502046683s
2021/05/10 00:00:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37164->10.242.163.157:8301: i/o timeout
2021/05/10 00:00:37 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:00:37 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:00:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:00:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:53338->10.242.163.138:8301: i/o timeout
2021/05/10 00:00:39 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:00:39 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.946619797s
2021/05/10 00:00:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:00:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:53366->10.242.163.138:8301: i/o timeout
2021/05/10 00:00:41 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:00:41 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 7.438290482s
2021/05/10 00:00:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:00:42 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/05/10 00:00:42 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:00:42 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:00:42 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/05/10 00:00:42 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:00:42 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:00:42 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/10 00:00:43 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/10 00:00:43 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:00:43 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/10 00:00:44 [ERR] raft: Failed to make RequestVote RPC to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.210.71:49494->10.242.163.157:8300: i/o timeout
2021/05/10 00:00:44 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.210.71:49498->10.242.163.157:8300: i/o timeout
2021/05/10 00:00:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:00:44 [WARN] raft: Rejecting vote request from 10.242.163.157:8300 since we have a leader: 10.242.210.71:8300
2021/05/10 00:00:45 [WARN] raft: AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300} rejected, sending older logs (next: 1738)
2021/05/10 00:00:47 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:00:47 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500319925s
2021/05/10 00:00:47 [INFO] raft: pipelining replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/10 00:00:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:00:53 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:00:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:00:57 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:00:58 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:00:58 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/10 00:00:59 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:01:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:53706->10.242.163.138:8301: i/o timeout
2021/05/10 00:01:02 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:01:03 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:01:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:04 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (2 peer confirmations)
2021/05/10 00:01:04 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:01:04 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/10 00:01:05 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-xplxm but other probes failed, network may be misconfigured
2021/05/10 00:01:09 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:01:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:12 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:01:12 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/10 00:01:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:53860->10.242.163.138:8301: i/o timeout
2021/05/10 00:01:14 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:01:15 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:01:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:18 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-xplxm but other probes failed, network may be misconfigured
2021/05/10 00:01:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37798->10.242.163.157:8301: i/o timeout
2021/05/10 00:01:19 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:01:19 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500292859s
2021/05/10 00:01:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:53964->10.242.163.138:8301: i/o timeout
2021/05/10 00:01:21 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:01:21 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 5.000214469s
2021/05/10 00:01:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:23 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/10 00:01:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:54024->10.242.163.138:8301: i/o timeout
2021/05/10 00:01:25 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:01:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:26 [ERR] raft: Failed to pipeline AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: write tcp 10.242.210.71:49662->10.242.163.157:8300: use of closed network connection
2021/05/10 00:01:26 [INFO] raft: aborting pipeline replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/10 00:01:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37906->10.242.163.157:8301: i/o timeout
2021/05/10 00:01:27 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:01:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:54086->10.242.163.138:8301: i/o timeout
2021/05/10 00:01:29 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:01:29 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500292454s
2021/05/10 00:01:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:30 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/05/10 00:01:30 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:01:30 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:01:30 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/05/10 00:01:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:37974->10.242.163.157:8301: i/o timeout
2021/05/10 00:01:31 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:01:31 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.923047441s
2021/05/10 00:01:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:34 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 7.40259612s
2021/05/10 00:01:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:54146->10.242.163.138:8301: i/o timeout
2021/05/10 00:01:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:01:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:33222->10.242.163.157:8302: i/o timeout
2021/05/10 00:01:36 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:01:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:54170->10.242.163.138:8301: i/o timeout
2021/05/10 00:01:36 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:01:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:36 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (1 peer confirmations)
2021/05/10 00:01:36 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:01:36 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/10 00:01:36 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.210.71:49508->10.242.163.157:8300: i/o timeout
2021/05/10 00:01:36 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:01:36 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: read tcp 10.242.210.71:50294->10.242.163.157:8300: i/o timeout
2021/05/10 00:01:37 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:01:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:33308->10.242.163.157:8302: i/o timeout
2021/05/10 00:01:46 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:01:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:46 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-xplxm 10.242.163.138:8301
2021/05/10 00:01:46 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.210.71:50436->10.242.163.157:8300: i/o timeout
2021/05/10 00:01:46 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:01:47 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: read tcp 10.242.210.71:50456->10.242.163.157:8300: i/o timeout
2021/05/10 00:01:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:48 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:01:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:33518->10.242.163.157:8302: i/o timeout
2021/05/10 00:01:56 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:01:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:56 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.210.71:50582->10.242.163.157:8300: i/o timeout
2021/05/10 00:01:56 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:01:56 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:01:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:01:58 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500290826s
2021/05/10 00:01:59 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:02:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:01 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500259444s
2021/05/10 00:02:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:04 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:02:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:04 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:02:05 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:02:05 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:02:05 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/10 00:02:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:38476->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:06 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:02:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:06 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.210.71:50726->10.242.163.157:8300: i/o timeout
2021/05/10 00:02:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:38514->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:08 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:02:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:08 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.501993656s
2021/05/10 00:02:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:38560->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:10 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:02:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:33748->10.242.163.157:8302: i/o timeout
2021/05/10 00:02:11 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:02:11 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.950710646s
2021/05/10 00:02:11 [ERR] memberlist: failed to receive: read tcp 10.242.210.71:8301->10.242.163.157:44092: i/o timeout from=10.242.163.157:44092
2021/05/10 00:02:11 [ERR] memberlist: Failed to send error: write tcp 10.242.210.71:8301->10.242.163.157:44092: i/o timeout from=10.242.163.157:44092
2021/05/10 00:02:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:38586->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:12 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:02:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:13 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 7.45565299s
2021/05/10 00:02:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:38604->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:14 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:02:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:38640->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:16 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:02:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:16 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.210.71:50618->10.242.163.157:8300: i/o timeout
2021/05/10 00:02:17 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: read tcp 10.242.210.71:50882->10.242.163.157:8300: i/o timeout
2021/05/10 00:02:17 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/05/10 00:02:17 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:02:17 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:02:17 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/05/10 00:02:17 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:02:17 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/10 00:02:17 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:02:18 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/10 00:02:18 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:02:18 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:02:18 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/10 00:02:18 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/05/10 00:02:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:22 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.503435723s
2021/05/10 00:02:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:38750->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:23 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:02:25 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:02:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:27 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.210.71:51038->10.242.163.157:8300: i/o timeout
2021/05/10 00:02:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:54944->10.242.163.138:8301: i/o timeout
2021/05/10 00:02:27 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/05/10 00:02:27 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:02:27 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:02:27 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/05/10 00:02:27 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:02:27 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:02:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:38824->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:28 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:02:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:28 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500329388s
2021/05/10 00:02:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:31 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:02:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:55036->10.242.163.138:8301: i/o timeout
2021/05/10 00:02:32 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:02:32 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:02:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:33 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:02:33 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:02:33 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/10 00:02:34 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-xplxm but other probes failed, network may be misconfigured
2021/05/10 00:02:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:38958->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:37 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:02:37 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.210.71:51058->10.242.163.157:8300: i/o timeout
2021/05/10 00:02:37 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:02:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:55110->10.242.163.138:8301: i/o timeout
2021/05/10 00:02:40 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:02:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:41 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500180998s
2021/05/10 00:02:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:43 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/10 00:02:43 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.983165592s
2021/05/10 00:02:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:55196->10.242.163.138:8301: i/o timeout
2021/05/10 00:02:44 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:02:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:55240->10.242.163.138:8301: i/o timeout
2021/05/10 00:02:47 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:02:47 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.210.71:51194->10.242.163.157:8300: i/o timeout
2021/05/10 00:02:47 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500271872s
2021/05/10 00:02:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:48 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (2 peer confirmations)
2021/05/10 00:02:48 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:02:48 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/10 00:02:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:39118->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:49 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:02:50 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.953063888s
2021/05/10 00:02:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:39172->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:51 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:02:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:52 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 7.427648543s
2021/05/10 00:02:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:39184->10.242.163.157:8301: i/o timeout
2021/05/10 00:02:53 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:02:53 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/05/10 00:02:53 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:02:53 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:02:53 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/05/10 00:02:53 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:02:54 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:02:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:55 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:02:55 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:02:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:34422->10.242.163.157:8302: i/o timeout
2021/05/10 00:02:56 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:02:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:02:56 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-xplxm 10.242.163.138:8301
2021/05/10 00:02:57 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500398032s
2021/05/10 00:02:58 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.210.71:51510->10.242.163.157:8300: i/o timeout
2021/05/10 00:02:58 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:02:58 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:02:58 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/10 00:02:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:39312->10.242.163.157:8301: i/o timeout
2021/05/10 00:03:01 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:03:01 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.501976316s
2021/05/10 00:03:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:39330->10.242.163.157:8301: i/o timeout
2021/05/10 00:03:03 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:03:04 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.981433896s
2021/05/10 00:03:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:39356->10.242.163.157:8301: i/o timeout
2021/05/10 00:03:06 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:03:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:06 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 7.436827216s
2021/05/10 00:03:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:39424->10.242.163.157:8301: i/o timeout
2021/05/10 00:03:09 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:03:09 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.210.71:51364->10.242.163.157:8300: i/o timeout
2021/05/10 00:03:09 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: read tcp 10.242.210.71:51678->10.242.163.157:8300: i/o timeout
2021/05/10 00:03:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:39446->10.242.163.157:8301: i/o timeout
2021/05/10 00:03:11 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:03:12 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:03:12 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:03:12 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/05/10 00:03:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:39484->10.242.163.157:8301: i/o timeout
2021/05/10 00:03:13 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:03:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:14 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-xplxm)
2021/05/10 00:03:14 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:03:14 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:03:14 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/10 00:03:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:39528->10.242.163.157:8301: i/o timeout
2021/05/10 00:03:17 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:03:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:39604->10.242.163.157:8301: i/o timeout
2021/05/10 00:03:20 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:03:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:21 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500257221s
2021/05/10 00:03:22 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.210.71:51854->10.242.163.157:8300: i/o timeout
2021/05/10 00:03:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:39628->10.242.163.157:8301: i/o timeout
2021/05/10 00:03:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:03:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:24 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.931055399s
2021/05/10 00:03:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:39642->10.242.163.157:8301: i/o timeout
2021/05/10 00:03:25 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:03:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:39706->10.242.163.157:8301: i/o timeout
2021/05/10 00:03:28 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:03:28 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/05/10 00:03:28 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:03:28 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:03:28 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/05/10 00:03:28 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500648664s
2021/05/10 00:03:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:30 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.945373577s
2021/05/10 00:03:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:34936->10.242.163.157:8302: i/o timeout
2021/05/10 00:03:31 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:03:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:33 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 7.412415571s
2021/05/10 00:03:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:36 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/10 00:03:36 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.163.138:60758
2021/05/10 00:03:36 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-xplxm 10.242.163.138:8301
2021/05/10 00:03:36 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:03:37 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.210.71:52074->10.242.163.157:8300: i/o timeout
2021/05/10 00:03:37 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:03:37 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:03:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:38 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:03:39 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:03:40 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:03:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:35084->10.242.163.157:8302: i/o timeout
2021/05/10 00:03:41 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:03:42 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:03:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:43 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:03:44 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:03:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:45 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:03:46 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:03:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:46 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:03:47 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:03:47 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:03:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:49 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500173864s
2021/05/10 00:03:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:35230->10.242.163.157:8302: i/o timeout
2021/05/10 00:03:51 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:03:51 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.905956924s
2021/05/10 00:03:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:54 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 7.349856814s
2021/05/10 00:03:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:03:57 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.210.71:51834->10.242.163.157:8300: i/o timeout
2021/05/10 00:03:57 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: read tcp 10.242.210.71:52366->10.242.163.157:8300: i/o timeout
2021/05/10 00:03:57 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:03:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:35290->10.242.163.157:8302: i/o timeout
2021/05/10 00:04:01 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:04:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:04 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/10 00:04:04 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/05/10 00:04:05 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:04:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:07 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:04:07 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: read tcp 10.242.210.71:52518->10.242.163.157:8300: i/o timeout
2021/05/10 00:04:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:08 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:04:09 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:04:09 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/10 00:04:09 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:04:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:35514->10.242.163.157:8302: i/o timeout
2021/05/10 00:04:11 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:04:11 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500248126s
2021/05/10 00:04:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:56488->10.242.163.138:8301: i/o timeout
2021/05/10 00:04:12 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:04:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:13 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/10 00:04:13 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.163.157
2021/05/10 00:04:13 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/05/10 00:04:13 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.957563883s
2021/05/10 00:04:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:56502->10.242.163.138:8301: i/o timeout
2021/05/10 00:04:14 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:04:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:14 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:04:14 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/10 00:04:16 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 7.395368111s
2021/05/10 00:04:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:16 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-xplxm 10.242.163.138:8301
2021/05/10 00:04:17 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.210.71:52656->10.242.163.157:8300: i/o timeout
2021/05/10 00:04:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:18 [ERR] memberlist: failed to receive: read tcp 10.242.210.71:8301->10.242.163.157:45126: i/o timeout from=10.242.163.157:45126
2021/05/10 00:04:18 [ERR] memberlist: Failed to send error: write tcp 10.242.210.71:8301->10.242.163.157:45126: i/o timeout from=10.242.163.157:45126
2021/05/10 00:04:19 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.163.157
2021/05/10 00:04:19 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/05/10 00:04:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:20 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:04:21 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:04:21 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:04:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:22 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:04:24 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:04:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:35722->10.242.163.157:8302: i/o timeout
2021/05/10 00:04:26 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:04:26 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500312103s
2021/05/10 00:04:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:27 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:04:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:28 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.917869054s
2021/05/10 00:04:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:31 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 7.377603687s
2021/05/10 00:04:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:34 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: read tcp 10.242.210.71:52674->10.242.163.157:8300: i/o timeout
2021/05/10 00:04:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:34 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/05/10 00:04:35 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:04:35 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:04:35 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:04:35 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/10 00:04:36 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/05/10 00:04:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:40692->10.242.163.157:8301: i/o timeout
2021/05/10 00:04:37 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:04:37 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.210.71:52948->10.242.163.157:8300: i/o timeout
2021/05/10 00:04:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:39 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/10 00:04:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:40 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.501533029s
2021/05/10 00:04:42 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/10 00:04:43 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:04:43 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/10 00:04:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:50 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/10 00:04:51 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:04:51 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/10 00:04:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:40920->10.242.163.157:8301: i/o timeout
2021/05/10 00:04:52 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:04:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:04:55 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500247882s
2021/05/10 00:04:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:40978->10.242.163.157:8301: i/o timeout
2021/05/10 00:04:56 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:04:56 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (0 peer confirmations)
2021/05/10 00:04:56 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:04:56 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:04:56 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/05/10 00:04:56 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:04:56 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:04:56 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/10 00:04:56 [INFO] raft: pipelining replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/10 00:04:57 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-2wbhv 10.242.168.100
2021/05/10 00:04:57 [INFO] consul: member 'dev-consul-7467bf7b46-2wbhv' joined, marking health alive
2021/05/10 00:04:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:41060->10.242.163.157:8301: i/o timeout
2021/05/10 00:04:58 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:04:59 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:05:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:41138->10.242.163.157:8301: i/o timeout
2021/05/10 00:05:01 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:05:01 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:05:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:05:03 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500292119s
2021/05/10 00:05:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:05:04 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:05:04 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:05:04 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/05/10 00:05:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:41196->10.242.163.157:8301: i/o timeout
2021/05/10 00:05:05 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:05:05 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.933635999s
2021/05/10 00:05:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:36384->10.242.163.157:8302: i/o timeout
2021/05/10 00:05:06 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:05:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:05:07 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 7.349726313s
2021/05/10 00:05:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:05:09 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:05:09 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:05:09 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/10 00:05:09 [ERR] raft: Failed to pipeline AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: write tcp 10.242.210.71:53042->10.242.163.157:8300: use of closed network connection
2021/05/10 00:05:09 [INFO] raft: aborting pipeline replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/10 00:05:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:05:12 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500232336s
2021/05/10 00:05:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:05:13 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/05/10 00:05:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:05:14 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.965945896s
2021/05/10 00:05:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:41364->10.242.163.157:8301: i/o timeout
2021/05/10 00:05:16 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:05:17 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:05:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:05:18 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/05/10 00:05:18 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:05:18 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:05:18 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/05/10 00:05:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:41382->10.242.163.157:8301: i/o timeout
2021/05/10 00:05:19 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:05:19 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:05:19 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: read tcp 10.242.210.71:53244->10.242.163.157:8300: i/o timeout
2021/05/10 00:05:19 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:05:20 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:05:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:05:21 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500323105s
2021/05/10 00:05:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:05:24 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.992120576s
2021/05/10 00:05:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:05:24 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:05:24 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/10 00:05:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:05:26 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 7.393644923s
2021/05/10 00:05:28 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.157
2021/05/10 00:05:28 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:05:28 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/10 00:05:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:05:29 [INFO] raft: pipelining replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/10 00:05:37 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:05:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:57852->10.242.163.138:8301: i/o timeout
2021/05/10 00:05:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:05:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:57970->10.242.163.138:8301: i/o timeout
2021/05/10 00:05:44 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:05:45 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (2 peer confirmations)
2021/05/10 00:05:45 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:05:45 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/10 00:05:47 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:05:47 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/10 00:05:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:58054->10.242.163.138:8301: i/o timeout
2021/05/10 00:05:49 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:05:49 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:05:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.210.71:58154->10.242.163.138:8301: i/o timeout
2021/05/10 00:05:53 [INFO] memberlist: Suspect dev-consul-7467bf7b46-xplxm has failed, no acks received
2021/05/10 00:05:53 [INFO] memberlist: Marking dev-consul-7467bf7b46-xplxm as failed, suspect timeout reached (2 peer confirmations)
2021/05/10 00:05:53 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:05:53 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' failed, marking health critical
2021/05/10 00:05:53 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:05:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:05:56 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:05:56 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' joined, marking health alive
2021/05/10 00:06:10 [INFO] serf: EventMemberLeave: dev-consul-7467bf7b46-xplxm 10.242.163.138
2021/05/10 00:06:10 [INFO] consul: member 'dev-consul-7467bf7b46-xplxm' left, deregistering
2021/05/10 00:06:10 [INFO] raft: aborting pipeline replication to peer {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}
2021/05/10 00:06:10 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: EOF
2021/05/10 00:06:10 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: dial tcp ->10.242.163.157:8300: getsockopt: connection refused
2021/05/10 00:06:10 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: dial tcp ->10.242.163.157:8300: getsockopt: connection refused
2021/05/10 00:06:10 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: dial tcp ->10.242.163.157:8300: getsockopt: connection refused
2021/05/10 00:06:11 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: dial tcp ->10.242.163.157:8300: getsockopt: connection refused
2021/05/10 00:06:11 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: dial tcp ->10.242.163.157:8300: getsockopt: connection refused
2021/05/10 00:06:11 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: dial tcp ->10.242.163.157:8300: getsockopt: connection refused
2021/05/10 00:06:11 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: dial tcp ->10.242.163.157:8300: getsockopt: connection refused
2021/05/10 00:06:11 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: dial tcp ->10.242.163.157:8300: getsockopt: connection refused
2021/05/10 00:06:12 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: dial tcp ->10.242.163.157:8300: getsockopt: connection refused
2021/05/10 00:06:12 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:06:12 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: dial tcp ->10.242.163.157:8300: getsockopt: connection refused
2021/05/10 00:06:12 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: stream closed
2021/05/10 00:06:12 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: dial tcp ->10.242.163.157:8300: getsockopt: connection refused
2021/05/10 00:06:12 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: dial tcp ->10.242.163.157:8300: getsockopt: connection refused
2021/05/10 00:06:13 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 2.500181621s
2021/05/10 00:06:13 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:06:13 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: dial tcp ->10.242.163.157:8300: getsockopt: connection refused
2021/05/10 00:06:14 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: dial tcp ->10.242.163.157:8300: getsockopt: connection refused
2021/05/10 00:06:14 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.163.157:8300: getsockopt: connection refused
2021/05/10 00:06:14 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: dial tcp ->10.242.163.157:8300: getsockopt: connection refused
2021/05/10 00:06:15 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:06:15 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 4.993016325s
2021/05/10 00:06:15 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: dial tcp ->10.242.163.157:8300: getsockopt: connection refused
2021/05/10 00:06:16 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/05/10 00:06:16 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.163.157
2021/05/10 00:06:16 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.163.157:8300) (DC: dc1)
2021/05/10 00:06:16 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/05/10 00:06:16 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.163.157:8300: getsockopt: connection refused
2021/05/10 00:06:16 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:06:16 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: dial tcp ->10.242.163.157:8300: getsockopt: connection refused
2021/05/10 00:06:16 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:06:16 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: dial tcp ->10.242.163.157:8300: getsockopt: connection refused
2021/05/10 00:06:17 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/05/10 00:06:17 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:06:17 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/10 00:06:18 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:06:18 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: dial tcp ->10.242.163.157:8300: getsockopt: connection refused
2021/05/10 00:06:18 [WARN] raft: Failed to contact 19ce88fc-998f-ee2b-b2d5-6fafec370e52 in 7.491980412s
2021/05/10 00:06:18 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.163.157:8300: getsockopt: connection refused
2021/05/10 00:06:19 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:06:19 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/10 00:06:19 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:06:21 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:06:21 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:06:21 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:06:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:06:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:06:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:06:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:06:29 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: dial tcp ->10.242.163.157:8300: i/o timeout
2021/05/10 00:06:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:06:30 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.163.157:8300: i/o timeout
2021/05/10 00:06:31 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: dial tcp ->10.242.163.157:8300: i/o timeout
2021/05/10 00:06:33 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:06:33 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:06:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:06:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:06:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:06:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:06:42 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:06:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:06:42 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.163.157:8300: i/o timeout
2021/05/10 00:06:43 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: dial tcp ->10.242.163.157:8300: i/o timeout
2021/05/10 00:06:45 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:06:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:06:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:06:49 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:06:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:06:51 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/10 00:06:51 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.163.157
2021/05/10 00:06:51 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/05/10 00:06:52 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: dial tcp ->10.242.163.157:8300: i/o timeout
2021/05/10 00:06:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:06:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:06:54 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.163.157:8300: i/o timeout
2021/05/10 00:06:57 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/10 00:06:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/05/10 00:06:59 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: dial tcp ->10.242.163.157:8300: i/o timeout
2021/05/10 00:06:59 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.163.183
2021/05/10 00:06:59 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.163.183:8300) (DC: dc1)
2021/05/10 00:06:59 [INFO] raft: Updating configuration with AddNonvoter (15ab35b5-2009-76b1-9910-289d503db0c7, 10.242.163.183:8300) to [{Suffrage:Voter ID:b3d9ccb9-04c2-7b7e-2bea-552e392b6c00 Address:10.242.210.71:8300} {Suffrage:Voter ID:19ce88fc-998f-ee2b-b2d5-6fafec370e52 Address:10.242.163.157:8300} {Suffrage:Voter ID:6ff779d7-507c-f4ae-1d24-7621a2e1d40c Address:10.242.61.70:8300} {Suffrage:Nonvoter ID:15ab35b5-2009-76b1-9910-289d503db0c7 Address:10.242.163.183:8300}]
2021/05/10 00:06:59 [INFO] raft: Added peer 15ab35b5-2009-76b1-9910-289d503db0c7, starting replication
2021/05/10 00:06:59 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/10 00:06:59 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.163.157:8302 Theirs: 10.242.163.183:8302
2021/05/10 00:06:59 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.163.157:8302 and 10.242.163.183:8302 are claiming
2021/05/10 00:06:59 [WARN] raft: AppendEntries to {Nonvoter 15ab35b5-2009-76b1-9910-289d503db0c7 10.242.163.183:8300} rejected, sending older logs (next: 1)
2021/05/10 00:07:00 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.163.157:8302 Theirs: 10.242.163.183:8302
2021/05/10 00:07:00 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.163.157:8302 and 10.242.163.183:8302 are claiming
2021/05/10 00:07:00 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.163.157:8302 Theirs: 10.242.163.183:8302
2021/05/10 00:07:00 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.163.157:8302 and 10.242.163.183:8302 are claiming
2021/05/10 00:07:00 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.163.157:8302 Theirs: 10.242.163.183:8302
2021/05/10 00:07:00 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.163.157:8302 and 10.242.163.183:8302 are claiming
2021/05/10 00:07:00 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.163.157:8302 Theirs: 10.242.163.183:8302
2021/05/10 00:07:00 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.163.157:8302 and 10.242.163.183:8302 are claiming
2021/05/10 00:07:01 [INFO] raft: pipelining replication to peer {Nonvoter 15ab35b5-2009-76b1-9910-289d503db0c7 10.242.163.183:8300}
2021/05/10 00:07:02 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:07:04 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/05/10 00:07:06 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.163.157:8300: i/o timeout
2021/05/10 00:07:10 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:07:12 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: dial tcp ->10.242.163.157:8300: i/o timeout
2021/05/10 00:07:14 [INFO] autopilot: Promoting Server (ID: "15ab35b5-2009-76b1-9910-289d503db0c7" Address: "10.242.163.183:8300") to voter
2021/05/10 00:07:14 [INFO] raft: Updating configuration with AddStaging (15ab35b5-2009-76b1-9910-289d503db0c7, 10.242.163.183:8300) to [{Suffrage:Voter ID:b3d9ccb9-04c2-7b7e-2bea-552e392b6c00 Address:10.242.210.71:8300} {Suffrage:Voter ID:19ce88fc-998f-ee2b-b2d5-6fafec370e52 Address:10.242.163.157:8300} {Suffrage:Voter ID:6ff779d7-507c-f4ae-1d24-7621a2e1d40c Address:10.242.61.70:8300} {Suffrage:Voter ID:15ab35b5-2009-76b1-9910-289d503db0c7 Address:10.242.163.183:8300}]
2021/05/10 00:07:14 [INFO] autopilot: Attempting removal of stale Server (ID: "19ce88fc-998f-ee2b-b2d5-6fafec370e52" Address: "10.242.163.157:8300")
2021/05/10 00:07:14 [INFO] raft: Updating configuration with RemoveServer (19ce88fc-998f-ee2b-b2d5-6fafec370e52, ) to [{Suffrage:Voter ID:b3d9ccb9-04c2-7b7e-2bea-552e392b6c00 Address:10.242.210.71:8300} {Suffrage:Voter ID:6ff779d7-507c-f4ae-1d24-7621a2e1d40c Address:10.242.61.70:8300} {Suffrage:Voter ID:15ab35b5-2009-76b1-9910-289d503db0c7 Address:10.242.163.183:8300}]
2021/05/10 00:07:14 [INFO] raft: Removed peer 19ce88fc-998f-ee2b-b2d5-6fafec370e52, stopping replication after 2019
2021/05/10 00:07:20 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: dial tcp ->10.242.163.157:8300: i/o timeout
2021/05/10 00:07:22 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:07:31 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:07:32 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: dial tcp ->10.242.163.157:8300: i/o timeout
2021/05/10 00:07:41 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: dial tcp ->10.242.163.157:8300: i/o timeout
2021/05/10 00:07:42 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:07:51 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:07:52 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: dial tcp ->10.242.163.157:8300: i/o timeout
2021/05/10 00:08:01 [ERR] raft: Failed to heartbeat to 10.242.163.157:8300: dial tcp ->10.242.163.157:8300: i/o timeout
2021/05/10 00:08:03 [WARN] Unable to get address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52, using fallback address 10.242.163.157:8300: Could not find address for server id 19ce88fc-998f-ee2b-b2d5-6fafec370e52
2021/05/10 00:08:12 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.163.183
2021/05/10 00:08:12 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/05/10 00:08:13 [ERR] raft: Failed to AppendEntries to {Voter 19ce88fc-998f-ee2b-b2d5-6fafec370e52 10.242.163.157:8300}: dial tcp ->10.242.163.157:8300: i/o timeout