By type
+ CONSUL_BIND=
+ [ -n ]
+ CONSUL_CLIENT=
+ [ -n ]
+ CONSUL_DATA_DIR=/consul/data
+ CONSUL_CONFIG_DIR=/consul/config
+ [ -n ]
+ [ a = - ]
+ [ agent = agent ]
+ shift
+ set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
+ [ consul = consul ]
+ stat -c %u /consul/data
+ id -u consul
+ [ 100 != 100 ]
+ stat -c %u /consul/config
+ id -u consul
+ [ 100 != 100 ]
+ [ ! -z ]
+ exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
bootstrap_expect > 0: expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
Version: 'v1.0.6'
Node ID: 'd0130513-a81e-5dda-0b84-625a0a08dfdb'
Node name: 'dev-consul-server-0'
Datacenter: 'dc1' (Segment: '')
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.242.209.68 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2021/04/13 10:41:39 [INFO] raft: Initial configuration (index=0): []
2021/04/13 10:41:39 [INFO] raft: Node at 10.242.209.68:8300 [Follower] entering Follower state (Leader: "")
2021/04/13 10:41:39 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.209.68
2021/04/13 10:41:39 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.209.68
2021/04/13 10:41:39 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/04/13 10:41:39 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2021/04/13 10:41:39 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2021/04/13 10:41:39 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.209.68:8300) (DC: dc1)
2021/04/13 10:41:39 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
2021/04/13 10:41:39 [INFO] agent: started state syncer
2021/04/13 10:41:39 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer
2021/04/13 10:41:39 [INFO] agent: Joining LAN cluster...
2021/04/13 10:41:39 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/04/13 10:41:39 [WARN] memberlist: Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
2021/04/13 10:41:39 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
2021/04/13 10:41:40 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/04/13 10:41:40 [INFO] agent: (LAN) joined: 0 Err: 3 error(s) occurred:
* Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/04/13 10:41:40 [WARN] agent: Join LAN failed: , retrying in 30s
2021/04/13 10:41:46 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/13 10:41:47 [WARN] raft: no known peers, aborting election
==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=12d6771a-0d5d-90f8-dc70-945841e94361&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2021/04/13 10:42:02 [ERR] agent: Coordinate update error: No cluster leader
2021/04/13 10:42:10 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/04/13 10:42:10 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
2021/04/13 10:42:10 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/04/13 10:42:10 [INFO] agent: (LAN) joined: 1 Err:
2021/04/13 10:42:10 [INFO] agent: Join LAN completed. Synced with 1 initial agents
2021/04/13 10:42:15 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/13 10:42:19 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.214.76
2021/04/13 10:42:19 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 10:42:19 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.214.76:8300) (DC: dc1)
2021/04/13 10:42:19 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.214.76
2021/04/13 10:42:19 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/13 10:42:21 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 10:42:21 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 10:42:21 [INFO] consul: Existing Raft peers reported by dev-consul-server-2, disabling bootstrap mode
2021/04/13 10:42:21 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 10:42:21 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 10:42:28 [DEBUG] raft-net: 10.242.209.68:8300 accepted connection from: 10.242.213.219:48180
2021/04/13 10:42:28 [DEBUG] raft-net: 10.242.209.68:8300 accepted connection from: 10.242.213.219:48182
2021/04/13 10:42:28 [WARN] raft: Failed to get previous log: 1 log not found (last: 0)
2021/04/13 10:42:28 [INFO] consul: New leader elected: dev-consul-server-2
2021/04/13 10:42:31 [INFO] agent: Synced node info
2021/04/13 11:01:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:32970->10.242.213.207:8301: i/o timeout
2021/04/13 11:01:27 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:01:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:33024->10.242.213.207:8301: i/o timeout
2021/04/13 11:01:31 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:01:31 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:01:31 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:01:34 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:01:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:55530->10.242.213.219:8301: i/o timeout
2021/04/13 11:01:37 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:01:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:33130->10.242.213.207:8301: i/o timeout
2021/04/13 11:01:39 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:01:40 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:01:40 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:01:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:33142->10.242.213.207:8301: i/o timeout
2021/04/13 11:01:41 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:01:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:55600->10.242.213.219:8301: i/o timeout
2021/04/13 11:01:43 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:01:44 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:01:44 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-8cwfv)
2021/04/13 11:01:46 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/13 11:01:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:33254->10.242.213.207:8301: i/o timeout
2021/04/13 11:01:49 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:01:52 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/13 11:01:54 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-8cwfv but other probes failed, network may be misconfigured
2021/04/13 11:01:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:55760->10.242.213.219:8301: i/o timeout
2021/04/13 11:01:56 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:01:58 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-8cwfv but other probes failed, network may be misconfigured
2021/04/13 11:01:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:55796->10.242.213.219:8301: i/o timeout
2021/04/13 11:01:59 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:02:00 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:02:00 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:02:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:33408->10.242.213.207:8301: i/o timeout
2021/04/13 11:02:01 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:02:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:55850->10.242.213.219:8301: i/o timeout
2021/04/13 11:02:03 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:02:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:51486->10.242.213.219:8302: i/o timeout
2021/04/13 11:02:04 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:02:04 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:02:04 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:02:04 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:02:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:55872->10.242.213.219:8301: i/o timeout
2021/04/13 11:02:05 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:02:07 [WARN] raft: Rejecting vote request from 10.242.214.76:8300 since we have a leader: 10.242.213.219:8300
2021/04/13 11:02:07 [DEBUG] raft-net: 10.242.209.68:8300 accepted connection from: 10.242.214.76:37572
2021/04/13 11:02:09 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.213.219:8301
2021/04/13 11:02:11 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.110.209:35198
2021/04/13 11:02:12 [ERR] raft-net: Failed to decode incoming command: read tcp 10.242.209.68:8300->10.242.213.219:48180: read: connection reset by peer
2021/04/13 11:02:13 [WARN] raft: Rejecting vote request from 10.242.214.76:8300 since we have a leader: 10.242.213.219:8300
2021/04/13 11:02:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:51600->10.242.213.219:8302: i/o timeout
2021/04/13 11:02:14 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:02:17 [WARN] raft: Heartbeat timeout from "10.242.213.219:8300" reached, starting election
2021/04/13 11:02:17 [INFO] raft: Node at 10.242.209.68:8300 [Candidate] entering Candidate state in term 3
2021/04/13 11:02:17 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:02:17 [INFO] raft: Node at 10.242.209.68:8300 [Follower] entering Follower state (Leader: "")
2021/04/13 11:02:18 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:02:18 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:02:20 [INFO] consul: New leader elected: dev-consul-server-1
2021/04/13 11:02:21 [DEBUG] raft-net: 10.242.209.68:8300 accepted connection from: 10.242.214.76:37838
2021/04/13 11:02:22 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/13 11:02:22 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:02:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:56120->10.242.213.219:8301: i/o timeout
2021/04/13 11:02:24 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:02:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:33726->10.242.213.207:8301: i/o timeout
2021/04/13 11:02:27 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:02:27 [ERR] raft: Failed to make RequestVote RPC to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.209.68:59618->10.242.213.219:8300: i/o timeout
2021/04/13 11:02:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:56168->10.242.213.219:8301: i/o timeout
2021/04/13 11:02:29 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:02:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:56222->10.242.213.219:8301: i/o timeout
2021/04/13 11:02:32 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:02:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:33812->10.242.213.207:8301: i/o timeout
2021/04/13 11:02:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:02:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:33848->10.242.213.207:8301: i/o timeout
2021/04/13 11:02:36 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:02:36 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:02:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:56280->10.242.213.219:8301: i/o timeout
2021/04/13 11:02:38 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:02:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:56316->10.242.213.219:8301: i/o timeout
2021/04/13 11:02:40 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:02:40 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:02:40 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:02:40 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:02:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:51992->10.242.213.219:8302: i/o timeout
2021/04/13 11:02:44 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:02:49 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:02:59 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/13 11:03:03 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/13 11:03:08 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:03:08 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:03:08 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:03:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:56720->10.242.213.219:8301: i/o timeout
2021/04/13 11:03:10 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:03:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:56772->10.242.213.219:8301: i/o timeout
2021/04/13 11:03:13 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:03:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:56806->10.242.213.219:8301: i/o timeout
2021/04/13 11:03:16 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:03:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:56900->10.242.213.219:8301: i/o timeout
2021/04/13 11:03:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:03:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:34496->10.242.213.207:8301: i/o timeout
2021/04/13 11:03:25 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:03:28 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/13 11:03:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:34558->10.242.213.207:8301: i/o timeout
2021/04/13 11:03:30 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:03:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:57032->10.242.213.219:8301: i/o timeout
2021/04/13 11:03:32 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:03:33 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:03:33 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:03:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:52676->10.242.213.219:8302: i/o timeout
2021/04/13 11:03:34 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:03:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:34640->10.242.213.207:8301: i/o timeout
2021/04/13 11:03:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:03:35 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:03:35 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:03:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:34648->10.242.213.207:8301: i/o timeout
2021/04/13 11:03:36 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:03:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:52796->10.242.213.219:8302: i/o timeout
2021/04/13 11:03:44 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:03:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:52850->10.242.213.219:8302: i/o timeout
2021/04/13 11:03:54 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:03:55 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:03:55 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:03:58 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/13 11:03:58 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/13 11:04:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:57350->10.242.213.219:8301: i/o timeout
2021/04/13 11:04:01 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:04:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:57396->10.242.213.219:8301: i/o timeout
2021/04/13 11:04:03 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:04:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:57420->10.242.213.219:8301: i/o timeout
2021/04/13 11:04:05 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:04:08 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:04:08 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:04:08 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:04:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:57456->10.242.213.219:8301: i/o timeout
2021/04/13 11:04:09 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:04:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:53146->10.242.213.219:8302: i/o timeout
2021/04/13 11:04:14 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:04:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:53204->10.242.213.219:8302: i/o timeout
2021/04/13 11:04:24 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:04:29 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.213.219:8301
2021/04/13 11:04:34 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:04:35 [ERR] yamux: keepalive failed: session shutdown
2021/04/13 11:04:35 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:04:35 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:04:35 [WARN] raft: Rejecting vote request from 10.242.213.219:8300 since we have a leader: 10.242.214.76:8300
2021/04/13 11:04:35 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/13 11:05:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:36288->10.242.213.207:8301: i/o timeout
2021/04/13 11:05:41 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:05:42 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-8cwfv)
2021/04/13 11:05:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:36362->10.242.213.207:8301: i/o timeout
2021/04/13 11:05:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:58822->10.242.213.219:8301: i/o timeout
2021/04/13 11:05:50 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:05:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:58868->10.242.213.219:8301: i/o timeout
2021/04/13 11:05:53 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:05:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:58952->10.242.213.219:8301: i/o timeout
2021/04/13 11:05:58 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:01 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-8cwfv but other probes failed, network may be misconfigured
2021/04/13 11:06:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:59024->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:02 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:36632->10.242.213.207:8301: i/o timeout
2021/04/13 11:06:04 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:06:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:36638->10.242.213.207:8301: i/o timeout
2021/04/13 11:06:06 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:06:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:59082->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:08 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:54734->10.242.213.219:8302: i/o timeout
2021/04/13 11:06:09 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:06:09 [ERR] memberlist: Push/Pull with dev-consul-7467bf7b46-8cwfv failed: read tcp 10.242.209.68:36556->10.242.213.207:8301: i/o timeout
2021/04/13 11:06:09 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:06:09 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:06:09 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:06:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:59116->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:10 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:11 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:06:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:36722->10.242.213.207:8301: i/o timeout
2021/04/13 11:06:12 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:06:15 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:06:15 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:06:15 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/13 11:06:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:59232->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:18 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:59244->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:20 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:59296->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:59320->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:25 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:59362->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:28 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:29 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:06:29 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:06:29 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:06:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:59378->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:30 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:31 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:06:31 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:06:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:59438->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:33 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:59456->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:35 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:35 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:06:37 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:06:37 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:06:37 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:06:37 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:06:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:59508->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:38 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:55142->10.242.213.219:8302: i/o timeout
2021/04/13 11:06:39 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:06:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:37118->10.242.213.207:8301: i/o timeout
2021/04/13 11:06:41 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:06:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:37176->10.242.213.207:8301: i/o timeout
2021/04/13 11:06:44 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:06:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:37198->10.242.213.207:8301: i/o timeout
2021/04/13 11:06:46 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:06:46 [ERR] memberlist: failed to receive: read tcp 10.242.209.68:8301->10.242.213.207:60294: i/o timeout from=10.242.213.207:60294
2021/04/13 11:06:46 [ERR] memberlist: Failed to send error: write tcp 10.242.209.68:8301->10.242.213.207:60294: i/o timeout from=10.242.213.207:60294
2021/04/13 11:06:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:37212->10.242.213.207:8301: i/o timeout
2021/04/13 11:06:48 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:06:49 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/13 11:06:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:37246->10.242.213.207:8301: i/o timeout
2021/04/13 11:06:50 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:06:50 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:06:50 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:06:52 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:06:52 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:06:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:59762->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:56 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:59796->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:58 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:07:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:59838->10.242.213.219:8301: i/o timeout
2021/04/13 11:07:01 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:07:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:55502->10.242.213.219:8302: i/o timeout
2021/04/13 11:07:04 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:07:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:59884->10.242.213.219:8301: i/o timeout
2021/04/13 11:07:04 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:07:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:59890->10.242.213.219:8301: i/o timeout
2021/04/13 11:07:06 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:07:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:59930->10.242.213.219:8301: i/o timeout
2021/04/13 11:07:08 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:07:09 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:07:09 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:07:09 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:07:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:59948->10.242.213.219:8301: i/o timeout
2021/04/13 11:07:10 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:07:10 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:07:10 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:07:16 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:07:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:37678->10.242.213.207:8301: i/o timeout
2021/04/13 11:07:20 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:07:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:37724->10.242.213.207:8301: i/o timeout
2021/04/13 11:07:23 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:07:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:37764->10.242.213.207:8301: i/o timeout
2021/04/13 11:07:26 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:07:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:55844->10.242.213.219:8302: i/o timeout
2021/04/13 11:07:29 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:07:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:60192->10.242.213.219:8301: i/o timeout
2021/04/13 11:07:29 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:07:30 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:07:30 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:07:30 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:07:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:60248->10.242.213.219:8301: i/o timeout
2021/04/13 11:07:32 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:07:33 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/13 11:07:33 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:07:33 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:07:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:60354->10.242.213.219:8301: i/o timeout
2021/04/13 11:07:39 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:07:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:38116->10.242.213.207:8301: i/o timeout
2021/04/13 11:07:51 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:07:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:60558->10.242.213.219:8301: i/o timeout
2021/04/13 11:07:54 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:07:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:60600->10.242.213.219:8301: i/o timeout
2021/04/13 11:07:57 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:07:58 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:07:58 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:07:59 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/13 11:07:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:38218->10.242.213.207:8301: i/o timeout
2021/04/13 11:07:59 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:08:00 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:08:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:60656->10.242.213.219:8301: i/o timeout
2021/04/13 11:08:01 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:08:03 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/13 11:08:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:60718->10.242.213.219:8301: i/o timeout
2021/04/13 11:08:04 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:08:04 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/13 11:08:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:60812->10.242.213.219:8301: i/o timeout
2021/04/13 11:08:11 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:08:14 [WARN] raft: Rejecting vote request from 10.242.213.219:8300 since we have a leader: 10.242.214.76:8300
2021/04/13 11:08:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:60922->10.242.213.219:8301: i/o timeout
2021/04/13 11:08:19 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:08:19 [WARN] raft: Heartbeat timeout from "10.242.214.76:8300" reached, starting election
2021/04/13 11:08:19 [INFO] raft: Node at 10.242.209.68:8300 [Candidate] entering Candidate state in term 6
2021/04/13 11:08:19 [INFO] raft: Election won. Tally: 2
2021/04/13 11:08:19 [INFO] raft: Node at 10.242.209.68:8300 [Leader] entering Leader state
2021/04/13 11:08:19 [INFO] raft: Added peer f589d1e2-bc2b-3068-9a75-b76c06ae0d2b, starting replication
2021/04/13 11:08:19 [INFO] raft: Added peer dfed536e-3d79-cd25-9127-f2cb8950f102, starting replication
2021/04/13 11:08:19 [INFO] consul: cluster leadership acquired
2021/04/13 11:08:19 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/13 11:08:19 [INFO] raft: pipelining replication to peer {Voter f589d1e2-bc2b-3068-9a75-b76c06ae0d2b 10.242.214.76:8300}
2021/04/13 11:08:20 [WARN] raft: Rejecting vote request from 10.242.213.219:8300 since we have a leader: 10.242.209.68:8300
2021/04/13 11:08:20 [ERR] raft: peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300} has newer term, stopping replication
2021/04/13 11:08:20 [INFO] raft: aborting pipeline replication to peer {Voter f589d1e2-bc2b-3068-9a75-b76c06ae0d2b 10.242.214.76:8300}
2021/04/13 11:08:20 [INFO] raft: Node at 10.242.209.68:8300 [Follower] entering Follower state (Leader: "")
2021/04/13 11:08:20 [INFO] consul: cluster leadership lost
2021/04/13 11:08:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:38566->10.242.213.207:8301: i/o timeout
2021/04/13 11:08:22 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:08:24 [WARN] consul.coordinate: Batch update failed: node is not the leader
2021/04/13 11:08:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:38598->10.242.213.207:8301: i/o timeout
2021/04/13 11:08:25 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:08:27 [ERR] agent: Coordinate update error: No cluster leader
2021/04/13 11:08:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:38624->10.242.213.207:8301: i/o timeout
2021/04/13 11:08:28 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:08:28 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/13 11:08:28 [INFO] raft: Node at 10.242.209.68:8300 [Candidate] entering Candidate state in term 7
2021/04/13 11:08:28 [INFO] raft: Election won. Tally: 2
2021/04/13 11:08:28 [INFO] raft: Node at 10.242.209.68:8300 [Leader] entering Leader state
2021/04/13 11:08:28 [INFO] raft: Added peer f589d1e2-bc2b-3068-9a75-b76c06ae0d2b, starting replication
2021/04/13 11:08:28 [INFO] raft: Added peer dfed536e-3d79-cd25-9127-f2cb8950f102, starting replication
2021/04/13 11:08:28 [INFO] consul: cluster leadership acquired
2021/04/13 11:08:28 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/13 11:08:28 [INFO] raft: pipelining replication to peer {Voter f589d1e2-bc2b-3068-9a75-b76c06ae0d2b 10.242.214.76:8300}
2021/04/13 11:08:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:56690->10.242.213.219:8302: i/o timeout
2021/04/13 11:08:30 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.501605122s
2021/04/13 11:08:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:32832->10.242.213.219:8301: i/o timeout
2021/04/13 11:08:31 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:08:31 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:08:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:08:33 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.953504s
2021/04/13 11:08:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:38726->10.242.213.207:8301: i/o timeout
2021/04/13 11:08:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:08:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:08:35 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:08:35 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:08:35 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' failed, marking health critical
2021/04/13 11:08:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:32918->10.242.213.219:8301: i/o timeout
2021/04/13 11:08:36 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:08:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:08:37 [ERR] raft: peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300} has newer term, stopping replication
2021/04/13 11:08:37 [INFO] raft: Node at 10.242.209.68:8300 [Follower] entering Follower state (Leader: "")
2021/04/13 11:08:37 [INFO] raft: aborting pipeline replication to peer {Voter f589d1e2-bc2b-3068-9a75-b76c06ae0d2b 10.242.214.76:8300}
2021/04/13 11:08:37 [INFO] consul: cluster leadership lost
2021/04/13 11:08:37 [WARN] raft: Rejecting vote request from 10.242.213.219:8300 since our last term is greater (7, 5)
2021/04/13 11:08:37 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:08:43 [INFO] consul: New leader elected: dev-consul-server-1
2021/04/13 11:08:43 [WARN] raft: Rejecting vote request from 10.242.213.219:8300 since we have a leader: 10.242.214.76:8300
2021/04/13 11:08:44 [DEBUG] raft-net: 10.242.209.68:8300 accepted connection from: 10.242.214.76:44662
2021/04/13 11:09:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:39182->10.242.213.207:8301: i/o timeout
2021/04/13 11:09:07 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:09:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:39206->10.242.213.207:8301: i/o timeout
2021/04/13 11:09:10 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:09:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:33508->10.242.213.219:8301: i/o timeout
2021/04/13 11:09:17 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:09:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:39342->10.242.213.207:8301: i/o timeout
2021/04/13 11:09:20 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:09:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:39410->10.242.213.207:8301: i/o timeout
2021/04/13 11:09:23 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:09:23 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:09:23 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:09:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:33710->10.242.213.219:8301: i/o timeout
2021/04/13 11:09:31 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:09:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:33740->10.242.213.219:8301: i/o timeout
2021/04/13 11:09:33 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:09:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:33782->10.242.213.219:8301: i/o timeout
2021/04/13 11:09:36 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:09:37 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:09:37 [ERR] memberlist: failed to receive: read tcp 10.242.209.68:8301->10.242.213.207:35440: i/o timeout from=10.242.213.207:35440
2021/04/13 11:09:37 [ERR] memberlist: Failed to send error: write tcp 10.242.209.68:8301->10.242.213.207:35440: i/o timeout from=10.242.213.207:35440
2021/04/13 11:09:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:33812->10.242.213.219:8301: i/o timeout
2021/04/13 11:09:39 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:09:39 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/13 11:09:43 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:09:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:33940->10.242.213.219:8301: i/o timeout
2021/04/13 11:09:47 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:09:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:39814->10.242.213.207:8301: i/o timeout
2021/04/13 11:09:51 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:09:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:39840->10.242.213.207:8301: i/o timeout
2021/04/13 11:09:54 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:09:55 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/13 11:09:57 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:09:57 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:09:57 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:10:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:34120->10.242.213.219:8301: i/o timeout
2021/04/13 11:10:00 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:10:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:40096->10.242.213.207:8301: i/o timeout
2021/04/13 11:10:11 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:10:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:40124->10.242.213.207:8301: i/o timeout
2021/04/13 11:10:13 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:10:16 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/13 11:10:16 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/13 11:10:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:40198->10.242.213.207:8301: i/o timeout
2021/04/13 11:10:19 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:10:19 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:10:22 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:10:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:34472->10.242.213.219:8301: i/o timeout
2021/04/13 11:11:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:41066->10.242.213.207:8301: i/o timeout
2021/04/13 11:11:20 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:11:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:41180->10.242.213.207:8301: i/o timeout
2021/04/13 11:11:28 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:11:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:35386->10.242.213.219:8301: i/o timeout
2021/04/13 11:11:31 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:11:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:41256->10.242.213.207:8301: i/o timeout
2021/04/13 11:11:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:11:34 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/13 11:11:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:41310->10.242.213.207:8301: i/o timeout
2021/04/13 11:11:37 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:11:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:35530->10.242.213.219:8301: i/o timeout
2021/04/13 11:11:40 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:11:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:41370->10.242.213.207:8301: i/o timeout
2021/04/13 11:11:43 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:11:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:41410->10.242.213.207:8301: i/o timeout
2021/04/13 11:11:46 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:11:48 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:11:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:41472->10.242.213.207:8301: i/o timeout
2021/04/13 11:11:49 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:11:59 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/13 11:12:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:35802->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:00 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:35850->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:03 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:59718->10.242.213.219:8302: i/o timeout
2021/04/13 11:12:04 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:12:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:35874->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:05 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:35900->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:07 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:35922->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:09 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:10 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:12:10 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:12:10 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:12:10 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:12:10 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:12:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:35952->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:11 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:35994->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:14 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:36000->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:16 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:16 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:12:17 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/13 11:12:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:36058->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:18 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:18 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:12:18 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:12:18 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:12:19 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:12:19 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:12:19 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/13 11:12:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:41920->10.242.213.207:8301: i/o timeout
2021/04/13 11:12:22 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:12:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:36148->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:25 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:41980->10.242.213.207:8301: i/o timeout
2021/04/13 11:12:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:42072->10.242.213.207:8301: i/o timeout
2021/04/13 11:12:32 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:12:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:36262->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:35 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:36312->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:38 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:60200->10.242.213.219:8302: i/o timeout
2021/04/13 11:12:39 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:12:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:42158->10.242.213.207:8301: i/o timeout
2021/04/13 11:12:40 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:12:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:36374->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:42 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:43 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:12:43 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:12:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:42228->10.242.213.207:8301: i/o timeout
2021/04/13 11:12:44 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:12:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:36418->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:46 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:46 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:12:46 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:12:46 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:12:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:60268->10.242.213.219:8302: i/o timeout
2021/04/13 11:12:49 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:12:54 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:12:54 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:12:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:36566->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:56 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:36602->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:59 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:13:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:36612->10.242.213.219:8301: i/o timeout
2021/04/13 11:13:01 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:13:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:36682->10.242.213.219:8301: i/o timeout
2021/04/13 11:13:04 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:13:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:36704->10.242.213.219:8301: i/o timeout
2021/04/13 11:13:06 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:13:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:36714->10.242.213.219:8301: i/o timeout
2021/04/13 11:13:08 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:13:10 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:13:10 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:13:10 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:13:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:36766->10.242.213.219:8301: i/o timeout
2021/04/13 11:13:11 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:13:12 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:13:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:60668->10.242.213.219:8302: i/o timeout
2021/04/13 11:13:14 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:13:14 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-8cwfv)
2021/04/13 11:13:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:42634->10.242.213.207:8301: i/o timeout
2021/04/13 11:13:15 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:13:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:42656->10.242.213.207:8301: i/o timeout
2021/04/13 11:13:18 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:13:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:42708->10.242.213.207:8301: i/o timeout
2021/04/13 11:13:21 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:13:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:60810->10.242.213.219:8302: i/o timeout
2021/04/13 11:13:24 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:13:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:42772->10.242.213.207:8301: i/o timeout
2021/04/13 11:13:24 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:13:24 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:13:24 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.213.219:52026
2021/04/13 11:13:25 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:13:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:42778->10.242.213.207:8301: i/o timeout
2021/04/13 11:13:26 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:13:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:60936->10.242.213.219:8302: i/o timeout
2021/04/13 11:13:34 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:13:34 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:13:34 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.213.207:60484
2021/04/13 11:13:38 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:13:38 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:13:39 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.213.219:8302
2021/04/13 11:13:42 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.213.219:8301
2021/04/13 11:13:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:60990->10.242.213.219:8302: i/o timeout
2021/04/13 11:13:44 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:13:49 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:14:06 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:14:06 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:14:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:33190->10.242.213.219:8302: i/o timeout
2021/04/13 11:14:14 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:14:22 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:14:24 [ERR] memberlist: failed to receive: read tcp 10.242.209.68:8302->10.242.213.219:55656: i/o timeout from=10.242.213.219:55656
2021/04/13 11:14:24 [ERR] memberlist: Failed to send error: write tcp 10.242.209.68:8302->10.242.213.219:55656: i/o timeout from=10.242.213.219:55656
2021/04/13 11:14:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:33386->10.242.213.219:8302: i/o timeout
2021/04/13 11:14:29 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:14:31 [ERR] memberlist: failed to receive: read tcp 10.242.209.68:8301->10.242.213.219:40552: i/o timeout from=10.242.213.219:40552
2021/04/13 11:14:31 [ERR] memberlist: Failed to send error: write tcp 10.242.209.68:8301->10.242.213.219:40552: i/o timeout from=10.242.213.219:40552
2021/04/13 11:14:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:33450->10.242.213.219:8302: i/o timeout
2021/04/13 11:14:39 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:14:42 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/13 11:14:42 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:14:42 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:15:02 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:15:06 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.213.219:8302
2021/04/13 11:15:42 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:16:06 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.213.219:8302
2021/04/13 11:16:22 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.213.219:8301
2021/04/13 11:16:25 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:16:25 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:16:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:39184->10.242.213.219:8301: i/o timeout
2021/04/13 11:16:27 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:16:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:39212->10.242.213.219:8301: i/o timeout
2021/04/13 11:16:30 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:16:31 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (0 peer confirmations)
2021/04/13 11:16:31 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:16:31 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:16:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:39260->10.242.213.219:8301: i/o timeout
2021/04/13 11:16:32 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:16:55 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:17:35 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:17:36 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.213.219:8302
2021/04/13 11:18:15 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:18:36 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.213.219:8302
2021/04/13 11:18:43 [WARN] raft: Rejecting vote request from 10.242.213.219:8300 since we have a leader: 10.242.214.76:8300
2021/04/13 11:18:55 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.213.219:8301
2021/04/13 11:18:59 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:19:16 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:19:16 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:19:22 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:19:22 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.213.207:33952
2021/04/13 11:19:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:37090->10.242.213.219:8302: i/o timeout
2021/04/13 11:19:29 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:19:35 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:19:41 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-pmks5 10.242.110.227
2021/04/13 11:19:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:37372->10.242.213.219:8302: i/o timeout
2021/04/13 11:19:44 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:19:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:37438->10.242.213.219:8302: i/o timeout
2021/04/13 11:19:54 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:19:59 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/13 11:19:59 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:19:59 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:20:06 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.213.219:8302
2021/04/13 11:20:13 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:20:13 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:20:14 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:20:14 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:20:15 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:20:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:38248->10.242.213.219:8302: i/o timeout
2021/04/13 11:20:19 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:20:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:42724->10.242.213.219:8301: i/o timeout
2021/04/13 11:20:19 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:20:19 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/13 11:20:19 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:20:19 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:20:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:42758->10.242.213.219:8301: i/o timeout
2021/04/13 11:20:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:20:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:38514->10.242.213.219:8302: i/o timeout
2021/04/13 11:20:34 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:20:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:38952->10.242.213.219:8302: i/o timeout
2021/04/13 11:20:49 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:20:49 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/13 11:20:49 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:20:49 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:20:55 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:20:59 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/13 11:21:13 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.213.219:8302
2021/04/13 11:21:19 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/13 11:21:19 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:21:19 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:21:19 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:21:19 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:21:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:43968->10.242.213.219:8301: i/o timeout
2021/04/13 11:21:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:21:25 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:21:25 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:21:25 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:21:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:39742->10.242.213.219:8302: i/o timeout
2021/04/13 11:21:34 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:21:39 [ERR] memberlist: failed to receive: read tcp 10.242.209.68:8301->10.242.213.207:42556: i/o timeout from=10.242.213.207:42556
2021/04/13 11:21:39 [ERR] memberlist: Failed to send error: write tcp 10.242.209.68:8301->10.242.213.207:42556: i/o timeout from=10.242.213.207:42556
2021/04/13 11:21:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:39902->10.242.213.219:8302: i/o timeout
2021/04/13 11:21:44 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:21:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:40042->10.242.213.219:8302: i/o timeout
2021/04/13 11:21:54 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:21:54 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:21:54 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.213.207:34906
2021/04/13 11:22:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:40176->10.242.213.219:8302: i/o timeout
2021/04/13 11:22:04 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:22:04 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/13 11:22:04 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:22:04 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:22:05 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.213.219:8301
2021/04/13 11:22:07 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:22:07 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:22:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:44644->10.242.213.219:8301: i/o timeout
2021/04/13 11:22:11 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:22:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:44678->10.242.213.219:8301: i/o timeout
2021/04/13 11:22:14 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:22:14 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:22:14 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:22:14 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:22:28 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:22:28 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:22:37 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:22:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:40654->10.242.213.219:8302: i/o timeout
2021/04/13 11:22:39 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:22:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:40730->10.242.213.219:8302: i/o timeout
2021/04/13 11:22:49 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:23:03 [ERR] memberlist: failed to receive: read tcp 10.242.209.68:8302->10.242.213.219:59120: i/o timeout from=10.242.213.219:59120
2021/04/13 11:23:03 [ERR] memberlist: Failed to send error: write tcp 10.242.209.68:8302->10.242.213.219:59120: i/o timeout from=10.242.213.219:59120
2021/04/13 11:23:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:41016->10.242.213.219:8302: i/o timeout
2021/04/13 11:23:04 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:23:08 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:23:08 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:23:18 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:23:19 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.213.219:8302
2021/04/13 11:23:19 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:23:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:51532->10.242.213.207:8301: i/o timeout
2021/04/13 11:23:22 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:23:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:51576->10.242.213.207:8301: i/o timeout
2021/04/13 11:23:26 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:23:26 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:23:26 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:23:49 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:23:59 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:24:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:52072->10.242.213.207:8301: i/o timeout
2021/04/13 11:24:01 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:24:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:52126->10.242.213.207:8301: i/o timeout
2021/04/13 11:24:05 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:24:05 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:24:05 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:24:05 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:24:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:52154->10.242.213.207:8301: i/o timeout
2021/04/13 11:24:08 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:24:09 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:24:09 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:24:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:52192->10.242.213.207:8301: i/o timeout
2021/04/13 11:24:10 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:24:11 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:24:12 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:24:12 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:24:13 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:24:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:46452->10.242.213.219:8301: i/o timeout
2021/04/13 11:24:15 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:24:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:52466->10.242.213.207:8301: i/o timeout
2021/04/13 11:24:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:52542->10.242.213.207:8301: i/o timeout
2021/04/13 11:24:33 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:24:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:46896->10.242.213.219:8301: i/o timeout
2021/04/13 11:24:44 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:24:47 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:24:47 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:24:47 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:24:47 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:24:47 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:24:48 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-8cwfv but other probes failed, network may be misconfigured
2021/04/13 11:24:48 [WARN] raft: Rejecting vote request from 10.242.213.219:8300 since we have a leader: 10.242.214.76:8300
2021/04/13 11:24:57 [INFO] consul: New leader elected: dev-consul-server-1
2021/04/13 11:24:57 [WARN] raft: Rejecting vote request from 10.242.213.219:8300 since we have a leader: 10.242.214.76:8300
2021/04/13 11:24:58 [DEBUG] raft-net: 10.242.209.68:8300 accepted connection from: 10.242.214.76:59684
2021/04/13 11:25:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.209.68:53094->10.242.213.207:8301: i/o timeout
2021/04/13 11:25:09 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:25:18 [INFO] serf: EventMemberLeave: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:25:20 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:25:31 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:25:31 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:25:31 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:25:32 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:25:58 [ERR] memberlist: Conflicting address for dev-consul-server-2. Mine: 10.242.213.219:8301 Theirs: 10.242.213.195:8301
2021/04/13 11:25:58 [WARN] serf: Name conflict for 'dev-consul-server-2' both 10.242.213.219:8301 and 10.242.213.195:8301 are claiming
2021/04/13 11:25:58 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.213.219:8302 Theirs: 10.242.213.195:8302
2021/04/13 11:25:58 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.213.219:8302 and 10.242.213.195:8302 are claiming
2021/04/13 11:25:59 [ERR] memberlist: Conflicting address for dev-consul-server-2. Mine: 10.242.213.219:8301 Theirs: 10.242.213.195:8301
2021/04/13 11:25:59 [WARN] serf: Name conflict for 'dev-consul-server-2' both 10.242.213.219:8301 and 10.242.213.195:8301 are claiming
2021/04/13 11:25:59 [ERR] memberlist: Conflicting address for dev-consul-server-2. Mine: 10.242.213.219:8301 Theirs: 10.242.213.195:8301
2021/04/13 11:25:59 [WARN] serf: Name conflict for 'dev-consul-server-2' both 10.242.213.219:8301 and 10.242.213.195:8301 are claiming
2021/04/13 11:25:59 [ERR] memberlist: Conflicting address for dev-consul-server-2. Mine: 10.242.213.219:8301 Theirs: 10.242.213.195:8301
2021/04/13 11:25:59 [WARN] serf: Name conflict for 'dev-consul-server-2' both 10.242.213.219:8301 and 10.242.213.195:8301 are claiming
2021/04/13 11:25:59 [ERR] memberlist: Conflicting address for dev-consul-server-2. Mine: 10.242.213.219:8301 Theirs: 10.242.213.195:8301
2021/04/13 11:25:59 [WARN] serf: Name conflict for 'dev-consul-server-2' both 10.242.213.219:8301 and 10.242.213.195:8301 are claiming
2021/04/13 11:25:59 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.213.219:8302 Theirs: 10.242.213.195:8302
2021/04/13 11:25:59 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.213.219:8302 and 10.242.213.195:8302 are claiming
2021/04/13 11:25:59 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.213.219:8302 Theirs: 10.242.213.195:8302
2021/04/13 11:25:59 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.213.219:8302 and 10.242.213.195:8302 are claiming
2021/04/13 11:25:59 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.213.219:8302 Theirs: 10.242.213.195:8302
2021/04/13 11:25:59 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.213.219:8302 and 10.242.213.195:8302 are claiming
2021/04/13 11:25:59 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.213.219:8302 Theirs: 10.242.213.195:8302
2021/04/13 11:25:59 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.213.219:8302 and 10.242.213.195:8302 are claiming
2021/04/13 11:26:02 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:26:26 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:26:26 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:26:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:26:49 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.213.219:8302
2021/04/13 11:26:51 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.195
2021/04/13 11:26:51 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.195:8300) (DC: dc1)
2021/04/13 11:27:58 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.213.195
2021/04/13 11:27:58 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"