By type
+ CONSUL_BIND=
+ [ -n ]
+ CONSUL_CLIENT=
+ [ -n ]
+ CONSUL_DATA_DIR=/consul/data
+ CONSUL_CONFIG_DIR=/consul/config
+ [ -n ]
+ [ a = - ]
+ [ agent = agent ]
+ shift
+ set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
+ [ consul = consul ]
+ stat -c %u /consul/data
+ id -u consul
+ [ 100 != 100 ]
+ stat -c %u /consul/config
+ id -u consul
+ [ 100 != 100 ]
+ [ ! -z ]
+ exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
bootstrap_expect > 0: expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
Version: 'v1.0.6'
Node ID: 'f589d1e2-bc2b-3068-9a75-b76c06ae0d2b'
Node name: 'dev-consul-server-1'
Datacenter: 'dc1' (Segment: '')
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.242.214.76 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2021/04/13 10:42:07 [INFO] raft: Initial configuration (index=0): []
2021/04/13 10:42:07 [INFO] raft: Node at 10.242.214.76:8300 [Follower] entering Follower state (Leader: "")
2021/04/13 10:42:07 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.214.76
2021/04/13 10:42:07 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.214.76
2021/04/13 10:42:07 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.214.76:8300) (DC: dc1)
2021/04/13 10:42:07 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2021/04/13 10:42:07 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2021/04/13 10:42:07 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
2021/04/13 10:42:07 [INFO] agent: started state syncer
2021/04/13 10:42:07 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/13 10:42:07 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer
2021/04/13 10:42:07 [INFO] agent: Joining LAN cluster...
2021/04/13 10:42:07 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/04/13 10:42:07 [WARN] memberlist: Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
2021/04/13 10:42:07 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
2021/04/13 10:42:07 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/04/13 10:42:07 [INFO] agent: (LAN) joined: 0 Err: 3 error(s) occurred:
* Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
* Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/04/13 10:42:07 [WARN] agent: Join LAN failed: , retrying in 30s
2021/04/13 10:42:15 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/13 10:42:15 [WARN] raft: no known peers, aborting election
==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=92744c6f-4a3d-ccb5-dd13-5212bc35889b&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2021/04/13 10:42:19 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 10:42:19 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.209.68
2021/04/13 10:42:19 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/04/13 10:42:19 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.209.68
2021/04/13 10:42:19 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.209.68:8300) (DC: dc1)
2021/04/13 10:42:21 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 10:42:21 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 10:42:21 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 10:42:21 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 10:42:21 [INFO] consul: Existing Raft peers reported by dev-consul-server-2, disabling bootstrap mode
2021/04/13 10:42:28 [DEBUG] raft-net: 10.242.214.76:8300 accepted connection from: 10.242.213.219:45980
2021/04/13 10:42:28 [WARN] raft: Failed to get previous log: 1 log not found (last: 0)
2021/04/13 10:42:28 [INFO] consul: New leader elected: dev-consul-server-2
2021/04/13 10:42:29 [INFO] agent: Synced node info
2021/04/13 10:42:29 [DEBUG] raft-net: 10.242.214.76:8300 accepted connection from: 10.242.213.219:45994
2021/04/13 10:42:37 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/04/13 10:42:37 [INFO] agent: (LAN) joined: 3 Err:
2021/04/13 10:42:37 [INFO] agent: Join LAN completed. Synced with 3 initial agents
2021/04/13 11:01:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:49942->10.242.213.207:8301: i/o timeout
2021/04/13 11:01:27 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:01:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:49980->10.242.213.207:8301: i/o timeout
2021/04/13 11:01:30 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:01:31 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:01:33 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/13 11:01:34 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:01:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:50040->10.242.213.207:8301: i/o timeout
2021/04/13 11:01:35 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:01:36 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-8cwfv)
2021/04/13 11:01:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:47278->10.242.213.219:8301: i/o timeout
2021/04/13 11:01:38 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:01:40 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:01:40 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:01:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:50122->10.242.213.207:8301: i/o timeout
2021/04/13 11:01:42 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:01:44 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:01:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:47386->10.242.213.219:8301: i/o timeout
2021/04/13 11:01:44 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:01:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:47470->10.242.213.219:8301: i/o timeout
2021/04/13 11:01:48 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:01:51 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-8cwfv but other probes failed, network may be misconfigured
2021/04/13 11:01:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:47612->10.242.213.219:8301: i/o timeout
2021/04/13 11:01:52 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:01:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:47650->10.242.213.219:8301: i/o timeout
2021/04/13 11:01:54 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:01:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:50458->10.242.213.207:8301: i/o timeout
2021/04/13 11:01:56 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:01:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:47732->10.242.213.219:8301: i/o timeout
2021/04/13 11:01:58 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:02:00 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:02:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:50556->10.242.213.207:8301: i/o timeout
2021/04/13 11:02:00 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:02:01 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:02:01 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.213.219:45920
2021/04/13 11:02:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:47822->10.242.213.219:8301: i/o timeout
2021/04/13 11:02:02 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:02:04 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:02:04 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:02:04 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:02:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:47874->10.242.213.219:8301: i/o timeout
2021/04/13 11:02:04 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:02:07 [WARN] raft: Heartbeat timeout from "10.242.213.219:8300" reached, starting election
2021/04/13 11:02:07 [INFO] raft: Node at 10.242.214.76:8300 [Candidate] entering Candidate state in term 3
2021/04/13 11:02:07 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:02:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:38464->10.242.213.219:8302: i/o timeout
2021/04/13 11:02:07 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:02:07 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:02:13 [WARN] raft: Election timeout reached, restarting election
2021/04/13 11:02:13 [INFO] raft: Node at 10.242.214.76:8300 [Candidate] entering Candidate state in term 4
2021/04/13 11:02:13 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:02:15 [ERR] http: Request PUT /v1/txn, error: No cluster leader from=10.242.209.72:56442
2021/04/13 11:02:17 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/13 11:02:17 [ERR] raft: Failed to make RequestVote RPC to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:40336->10.242.213.219:8300: i/o timeout
2021/04/13 11:02:17 [DEBUG] raft-net: 10.242.214.76:8300 accepted connection from: 10.242.209.68:53016
2021/04/13 11:02:17 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/13 11:02:18 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:02:18 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:02:20 [WARN] raft: Election timeout reached, restarting election
2021/04/13 11:02:20 [INFO] raft: Node at 10.242.214.76:8300 [Candidate] entering Candidate state in term 5
2021/04/13 11:02:20 [INFO] raft: Election won. Tally: 2
2021/04/13 11:02:20 [INFO] raft: Node at 10.242.214.76:8300 [Leader] entering Leader state
2021/04/13 11:02:20 [INFO] raft: Added peer dfed536e-3d79-cd25-9127-f2cb8950f102, starting replication
2021/04/13 11:02:20 [INFO] raft: Added peer d0130513-a81e-5dda-0b84-625a0a08dfdb, starting replication
2021/04/13 11:02:20 [INFO] consul: cluster leadership acquired
2021/04/13 11:02:20 [INFO] consul: New leader elected: dev-consul-server-1
2021/04/13 11:02:20 [INFO] raft: pipelining replication to peer {Voter d0130513-a81e-5dda-0b84-625a0a08dfdb 10.242.209.68:8300}
2021/04/13 11:02:20 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' failed, marking health critical
2021/04/13 11:02:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:48156->10.242.213.219:8301: i/o timeout
2021/04/13 11:02:20 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:02:22 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:02:22 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' joined, marking health alive
2021/04/13 11:02:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:48196->10.242.213.219:8301: i/o timeout
2021/04/13 11:02:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:02:22 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.501885691s
2021/04/13 11:02:23 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:02:23 [ERR] raft: Failed to make RequestVote RPC to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:40462->10.242.213.219:8300: i/o timeout
2021/04/13 11:02:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:02:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:51002->10.242.213.207:8301: i/o timeout
2021/04/13 11:02:24 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:02:25 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.94569647s
2021/04/13 11:02:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:02:26 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/13 11:02:27 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 7.400255518s
2021/04/13 11:02:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:02:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:51068->10.242.213.207:8301: i/o timeout
2021/04/13 11:02:28 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:02:30 [ERR] raft: Failed to make RequestVote RPC to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:40584->10.242.213.219:8300: i/o timeout
2021/04/13 11:02:30 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:40586->10.242.213.219:8300: i/o timeout
2021/04/13 11:02:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:02:31 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:40598->10.242.213.219:8300: i/o timeout
2021/04/13 11:02:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:48322->10.242.213.219:8301: i/o timeout
2021/04/13 11:02:31 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:02:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:02:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:51154->10.242.213.207:8301: i/o timeout
2021/04/13 11:02:33 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:02:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:02:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:48404->10.242.213.219:8301: i/o timeout
2021/04/13 11:02:35 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:02:36 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:02:36 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:02:36 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' failed, marking health critical
2021/04/13 11:02:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:02:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:38982->10.242.213.219:8302: i/o timeout
2021/04/13 11:02:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:02:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:02:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:48456->10.242.213.219:8301: i/o timeout
2021/04/13 11:02:38 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:02:40 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:02:40 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:02:40 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:02:40 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/13 11:02:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:02:40 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:40752->10.242.213.219:8300: i/o timeout
2021/04/13 11:02:40 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:02:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:48466->10.242.213.219:8301: i/o timeout
2021/04/13 11:02:40 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:02:42 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:40800->10.242.213.219:8300: i/o timeout
2021/04/13 11:02:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:02:43 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:02:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:02:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:02:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:39092->10.242.213.219:8302: i/o timeout
2021/04/13 11:02:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:02:47 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:02:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:02:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:02:50 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:40928->10.242.213.219:8300: i/o timeout
2021/04/13 11:02:50 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:02:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:02:53 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:40990->10.242.213.219:8300: i/o timeout
2021/04/13 11:02:54 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:02:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:02:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:02:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:39480->10.242.213.219:8302: i/o timeout
2021/04/13 11:02:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:02:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:00 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:41168->10.242.213.219:8300: i/o timeout
2021/04/13 11:03:00 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:03:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:02 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/13 11:03:03 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:03:03 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/13 11:03:05 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.516682458s
2021/04/13 11:03:05 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:03:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:07 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.906369726s
2021/04/13 11:03:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:08 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:03:08 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:03:08 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/13 11:03:08 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:03:08 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' joined, marking health alive
2021/04/13 11:03:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:10 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:41468->10.242.213.219:8300: i/o timeout
2021/04/13 11:03:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:49274->10.242.213.219:8301: i/o timeout
2021/04/13 11:03:10 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:03:11 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.505586665s
2021/04/13 11:03:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:13 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.992044465s
2021/04/13 11:03:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:49396->10.242.213.219:8301: i/o timeout
2021/04/13 11:03:13 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:03:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:16 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 7.465974453s
2021/04/13 11:03:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:52218->10.242.213.207:8301: i/o timeout
2021/04/13 11:03:16 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:03:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:20 [INFO] raft: pipelining replication to peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}
2021/04/13 11:03:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:49596->10.242.213.219:8301: i/o timeout
2021/04/13 11:03:21 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:03:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:22 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.514061111s
2021/04/13 11:03:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:52410->10.242.213.207:8301: i/o timeout
2021/04/13 11:03:23 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:03:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:25 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.913713455s
2021/04/13 11:03:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:49674->10.242.213.219:8301: i/o timeout
2021/04/13 11:03:25 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:03:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:27 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/13 11:03:29 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:03:29 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/13 11:03:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:52526->10.242.213.207:8301: i/o timeout
2021/04/13 11:03:29 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:03:30 [INFO] raft: aborting pipeline replication to peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}
2021/04/13 11:03:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:52594->10.242.213.207:8301: i/o timeout
2021/04/13 11:03:31 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:03:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:49830->10.242.213.219:8301: i/o timeout
2021/04/13 11:03:32 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:03:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:40374->10.242.213.219:8302: i/o timeout
2021/04/13 11:03:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:03:32 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.506227312s
2021/04/13 11:03:33 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:03:33 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:03:33 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:03:33 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/13 11:03:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:52628->10.242.213.207:8301: i/o timeout
2021/04/13 11:03:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:03:35 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:03:35 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:03:35 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' failed, marking health critical
2021/04/13 11:03:35 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.935959141s
2021/04/13 11:03:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:37 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 7.39644153s
2021/04/13 11:03:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:39 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:41282->10.242.213.219:8300: i/o timeout
2021/04/13 11:03:40 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:03:40 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:42212->10.242.213.219:8300: i/o timeout
2021/04/13 11:03:40 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:03:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:43 [ERR] memberlist: failed to receive: read tcp 10.242.214.76:8301->10.242.213.207:49276: i/o timeout from=10.242.213.207:49276
2021/04/13 11:03:43 [ERR] memberlist: Failed to send error: write tcp 10.242.214.76:8301->10.242.213.207:49276: i/o timeout from=10.242.213.207:49276
2021/04/13 11:03:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:45 [ERR] memberlist: failed to receive: read tcp 10.242.214.76:8301->10.242.213.219:54684: i/o timeout from=10.242.213.219:54684
2021/04/13 11:03:45 [ERR] memberlist: Failed to send error: write tcp 10.242.214.76:8301->10.242.213.219:54684: i/o timeout from=10.242.213.219:54684
2021/04/13 11:03:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:40642->10.242.213.219:8302: i/o timeout
2021/04/13 11:03:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:03:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:50 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:42376->10.242.213.219:8300: i/o timeout
2021/04/13 11:03:50 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:42386->10.242.213.219:8300: i/o timeout
2021/04/13 11:03:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:50 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:03:50 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:03:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:55 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:03:55 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:03:55 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/13 11:03:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:03:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:50252->10.242.213.219:8301: i/o timeout
2021/04/13 11:03:56 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:03:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:40814->10.242.213.219:8302: i/o timeout
2021/04/13 11:03:57 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:03:57 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:03:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:50278->10.242.213.219:8301: i/o timeout
2021/04/13 11:03:58 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:03:59 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:04:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:04:00 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:42550->10.242.213.219:8300: i/o timeout
2021/04/13 11:04:00 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.50035414s
2021/04/13 11:04:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:50316->10.242.213.219:8301: i/o timeout
2021/04/13 11:04:00 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:04:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:04:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:50376->10.242.213.219:8301: i/o timeout
2021/04/13 11:04:02 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:04:03 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.942283663s
2021/04/13 11:04:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:04:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:50402->10.242.213.219:8301: i/o timeout
2021/04/13 11:04:04 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:04:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:04:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:50426->10.242.213.219:8301: i/o timeout
2021/04/13 11:04:06 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:04:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:40986->10.242.213.219:8302: i/o timeout
2021/04/13 11:04:07 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:04:08 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:04:08 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:04:08 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:04:08 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/13 11:04:08 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:04:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:04:09 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:04:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:04:10 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:42744->10.242.213.219:8300: i/o timeout
2021/04/13 11:04:10 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:04:11 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:04:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:04:13 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.505114439s
2021/04/13 11:04:13 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:04:13 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.213.207:58242
2021/04/13 11:04:13 [ERR] memberlist: failed to receive: read tcp 10.242.214.76:8301->10.242.213.219:54910: i/o timeout from=10.242.213.219:54910
2021/04/13 11:04:13 [ERR] memberlist: Failed to send error: write tcp 10.242.214.76:8301->10.242.213.219:54910: i/o timeout from=10.242.213.219:54910
2021/04/13 11:04:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:04:15 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.921080705s
2021/04/13 11:04:16 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:04:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:41124->10.242.213.219:8302: i/o timeout
2021/04/13 11:04:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:04:18 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 7.358696273s
2021/04/13 11:04:18 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:04:19 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:04:19 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/13 11:04:20 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:42912->10.242.213.219:8300: i/o timeout
2021/04/13 11:04:20 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:04:21 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:04:21 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:42564->10.242.213.219:8300: i/o timeout
2021/04/13 11:04:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:04:22 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:04:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:04:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:04:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:41250->10.242.213.219:8302: i/o timeout
2021/04/13 11:04:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:04:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:04:28 [ERR] memberlist: Push/Pull with dev-consul-server-2.dc1 failed: read tcp 10.242.214.76:40852->10.242.213.219:8302: i/o timeout
2021/04/13 11:04:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:04:30 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:43072->10.242.213.219:8300: i/o timeout
2021/04/13 11:04:30 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:04:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:04:32 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:43174->10.242.213.219:8300: i/o timeout
2021/04/13 11:04:33 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:04:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:04:34 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:04:34 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' joined, marking health alive
2021/04/13 11:04:35 [WARN] raft: Rejecting vote request from 10.242.213.219:8300 since we have a leader: 10.242.214.76:8300
2021/04/13 11:04:35 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/13 11:04:35 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:04:35 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:04:35 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/13 11:04:35 [INFO] raft: pipelining replication to peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}
2021/04/13 11:05:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:54902->10.242.213.207:8301: i/o timeout
2021/04/13 11:05:39 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:05:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:52304->10.242.213.219:8301: i/o timeout
2021/04/13 11:05:50 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:05:51 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:05:51 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/13 11:05:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:05:53 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/13 11:05:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:55152->10.242.213.207:8301: i/o timeout
2021/04/13 11:05:55 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:05:56 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-8cwfv)
2021/04/13 11:05:59 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500277561s
2021/04/13 11:05:59 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:05:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:55192->10.242.213.207:8301: i/o timeout
2021/04/13 11:05:59 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:06:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:06:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:06:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:52502->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:02 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:06:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:55298->10.242.213.207:8301: i/o timeout
2021/04/13 11:06:04 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:06:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:52528->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:05 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:06 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.504905602s
2021/04/13 11:06:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:06:06 [INFO] raft: aborting pipeline replication to peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}
2021/04/13 11:06:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:43104->10.242.213.219:8302: i/o timeout
2021/04/13 11:06:07 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:06:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:55326->10.242.213.207:8301: i/o timeout
2021/04/13 11:06:07 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:06:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:06:09 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.514968505s
2021/04/13 11:06:09 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:06:09 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:06:09 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:06:09 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/13 11:06:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:55356->10.242.213.207:8301: i/o timeout
2021/04/13 11:06:09 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:06:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:06:11 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:06:11 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:06:11 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' failed, marking health critical
2021/04/13 11:06:11 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.982633004s
2021/04/13 11:06:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:06:14 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 7.400667139s
2021/04/13 11:06:14 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:06:14 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:06:14 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.213.207:59906
2021/04/13 11:06:14 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:06:14 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.213.219:39266
2021/04/13 11:06:14 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:43354->10.242.213.219:8300: i/o timeout
2021/04/13 11:06:15 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:06:15 [INFO] raft: pipelining replication to peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}
2021/04/13 11:06:15 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:06:15 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:06:15 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/13 11:06:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:52704->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:18 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:19 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500170627s
2021/04/13 11:06:19 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:06:20 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:06:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:52738->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:20 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:21 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.954613044s
2021/04/13 11:06:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:06:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:52772->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:23 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 7.402565436s
2021/04/13 11:06:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:06:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:52798->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:25 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:06:27 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:45086->10.242.213.219:8300: i/o timeout
2021/04/13 11:06:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:43376->10.242.213.219:8302: i/o timeout
2021/04/13 11:06:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:52828->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:27 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:28 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.501619963s
2021/04/13 11:06:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:06:29 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:06:29 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:06:29 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/13 11:06:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:52850->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:29 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:06:30 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.994648294s
2021/04/13 11:06:31 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:06:31 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:06:31 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/13 11:06:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:06:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:52916->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:32 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:33 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500293209s
2021/04/13 11:06:34 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:06:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:52926->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:34 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:36 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.981955844s
2021/04/13 11:06:36 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:06:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:52950->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:36 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:37 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:06:37 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:06:37 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:06:37 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:06:37 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' joined, marking health alive
2021/04/13 11:06:37 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/13 11:06:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:06:38 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 7.457833627s
2021/04/13 11:06:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:55758->10.242.213.207:8301: i/o timeout
2021/04/13 11:06:38 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:06:39 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:06:39 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:06:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:06:41 [INFO] raft: aborting pipeline replication to peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}
2021/04/13 11:06:41 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:06:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:55810->10.242.213.207:8301: i/o timeout
2021/04/13 11:06:41 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:06:42 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:06:43 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500251706s
2021/04/13 11:06:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:55842->10.242.213.207:8301: i/o timeout
2021/04/13 11:06:43 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:06:44 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:06:46 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.957226877s
2021/04/13 11:06:46 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:06:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:43652->10.242.213.219:8302: i/o timeout
2021/04/13 11:06:47 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:06:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:55888->10.242.213.207:8301: i/o timeout
2021/04/13 11:06:47 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:06:48 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:06:48 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 7.445943492s
2021/04/13 11:06:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:55938->10.242.213.207:8301: i/o timeout
2021/04/13 11:06:49 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:06:50 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:06:50 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:06:50 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' failed, marking health critical
2021/04/13 11:06:50 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:06:51 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:45448->10.242.213.219:8300: i/o timeout
2021/04/13 11:06:51 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:06:51 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:06:51 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-8cwfv but other probes failed, network may be misconfigured
2021/04/13 11:06:52 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/13 11:06:52 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:06:52 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:06:52 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/13 11:06:52 [INFO] raft: pipelining replication to peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}
2021/04/13 11:06:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:53336->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:55 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:06:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:53362->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:53428->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:58 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:06:59 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500267967s
2021/04/13 11:06:59 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:06:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:53440->10.242.213.219:8301: i/o timeout
2021/04/13 11:06:59 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:07:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:07:02 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:07:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:53500->10.242.213.219:8301: i/o timeout
2021/04/13 11:07:02 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:07:03 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500294185s
2021/04/13 11:07:04 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:07:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:53536->10.242.213.219:8301: i/o timeout
2021/04/13 11:07:04 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:07:05 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.985572753s
2021/04/13 11:07:06 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:07:07 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:45254->10.242.213.219:8300: i/o timeout
2021/04/13 11:07:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:44140->10.242.213.219:8302: i/o timeout
2021/04/13 11:07:07 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:07:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:53592->10.242.213.219:8301: i/o timeout
2021/04/13 11:07:07 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:07:07 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:07:08 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 7.467209543s
2021/04/13 11:07:08 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:07:09 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:07:09 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:07:09 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:07:09 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/13 11:07:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:53612->10.242.213.219:8301: i/o timeout
2021/04/13 11:07:09 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:07:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:07:10 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:07:10 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:07:10 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/13 11:07:11 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/13 11:07:13 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:07:15 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:07:16 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:07:16 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' joined, marking health alive
2021/04/13 11:07:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:56652->10.242.213.207:8301: i/o timeout
2021/04/13 11:07:20 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:07:21 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:07:22 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:07:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:56686->10.242.213.207:8301: i/o timeout
2021/04/13 11:07:23 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:07:24 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:07:24 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500182276s
2021/04/13 11:07:26 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:07:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:53924->10.242.213.219:8301: i/o timeout
2021/04/13 11:07:26 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:07:28 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:07:29 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500324825s
2021/04/13 11:07:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:53958->10.242.213.219:8301: i/o timeout
2021/04/13 11:07:30 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:07:30 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:07:30 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:07:30 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:07:30 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/13 11:07:30 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-8cwfv but other probes failed, network may be misconfigured
2021/04/13 11:07:31 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:07:31 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:07:32 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:07:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:44582->10.242.213.219:8302: i/o timeout
2021/04/13 11:07:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:07:32 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:07:33 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:07:33 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:07:33 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/13 11:07:37 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:07:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:54092->10.242.213.219:8301: i/o timeout
2021/04/13 11:07:37 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:07:38 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:07:40 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:07:40 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-8cwfv but other probes failed, network may be misconfigured
2021/04/13 11:07:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:56954->10.242.213.207:8301: i/o timeout
2021/04/13 11:07:42 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:07:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57016->10.242.213.207:8301: i/o timeout
2021/04/13 11:07:48 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:07:49 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:07:49 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/13 11:07:50 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:07:50 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500229846s
2021/04/13 11:07:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:54266->10.242.213.219:8301: i/o timeout
2021/04/13 11:07:51 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:07:52 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:07:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:44850->10.242.213.219:8302: i/o timeout
2021/04/13 11:07:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57098->10.242.213.207:8301: i/o timeout
2021/04/13 11:07:53 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:07:54 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:07:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57106->10.242.213.207:8301: i/o timeout
2021/04/13 11:07:55 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:07:56 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:07:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:54358->10.242.213.219:8301: i/o timeout
2021/04/13 11:07:57 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:07:58 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:07:58 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:07:58 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' failed, marking health critical
2021/04/13 11:07:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:54382->10.242.213.219:8301: i/o timeout
2021/04/13 11:07:59 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:08:00 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:08:00 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' joined, marking health alive
2021/04/13 11:08:00 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:08:01 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-8cwfv)
2021/04/13 11:08:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57208->10.242.213.207:8301: i/o timeout
2021/04/13 11:08:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:44990->10.242.213.219:8302: i/o timeout
2021/04/13 11:08:09 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:08:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:54512->10.242.213.219:8301: i/o timeout
2021/04/13 11:08:09 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:08:10 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:08:10 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500178005s
2021/04/13 11:08:12 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:08:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57338->10.242.213.207:8301: i/o timeout
2021/04/13 11:08:12 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:08:13 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.997301764s
2021/04/13 11:08:13 [WARN] raft: Rejecting vote request from 10.242.213.219:8300 since we have a leader: 10.242.214.76:8300
2021/04/13 11:08:14 [ERR] raft: peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300} has newer term, stopping replication
2021/04/13 11:08:14 [INFO] raft: aborting pipeline replication to peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}
2021/04/13 11:08:14 [INFO] raft: Node at 10.242.214.76:8300 [Follower] entering Follower state (Leader: "")
2021/04/13 11:08:14 [INFO] raft: aborting pipeline replication to peer {Voter d0130513-a81e-5dda-0b84-625a0a08dfdb 10.242.209.68:8300}
2021/04/13 11:08:14 [INFO] consul: cluster leadership lost
2021/04/13 11:08:14 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-8cwfv)
2021/04/13 11:08:14 [WARN] consul.coordinate: Batch update failed: node is not the leader
2021/04/13 11:08:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57422->10.242.213.207:8301: i/o timeout
2021/04/13 11:08:18 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:08:19 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/13 11:08:20 [DEBUG] raft-net: 10.242.214.76:8300 accepted connection from: 10.242.209.68:57928
2021/04/13 11:08:20 [WARN] raft: Rejecting vote request from 10.242.213.219:8300 since we have a leader: 10.242.209.68:8300
2021/04/13 11:08:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57508->10.242.213.207:8301: i/o timeout
2021/04/13 11:08:23 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:08:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:54740->10.242.213.219:8301: i/o timeout
2021/04/13 11:08:26 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:08:28 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/13 11:08:28 [DEBUG] raft-net: 10.242.214.76:8300 accepted connection from: 10.242.209.68:58050
2021/04/13 11:08:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57558->10.242.213.207:8301: i/o timeout
2021/04/13 11:08:28 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:08:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57570->10.242.213.207:8301: i/o timeout
2021/04/13 11:08:30 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:08:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:54832->10.242.213.219:8301: i/o timeout
2021/04/13 11:08:32 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:08:35 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:08:35 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:08:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:54874->10.242.213.219:8301: i/o timeout
2021/04/13 11:08:35 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:08:37 [WARN] raft: Rejecting vote request from 10.242.213.219:8300 since we have a leader: 10.242.209.68:8300
2021/04/13 11:08:37 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:08:43 [WARN] raft: Heartbeat timeout from "10.242.209.68:8300" reached, starting election
2021/04/13 11:08:43 [INFO] raft: Node at 10.242.214.76:8300 [Candidate] entering Candidate state in term 8
2021/04/13 11:08:43 [INFO] raft: Election won. Tally: 2
2021/04/13 11:08:43 [INFO] raft: Node at 10.242.214.76:8300 [Leader] entering Leader state
2021/04/13 11:08:43 [INFO] raft: Added peer dfed536e-3d79-cd25-9127-f2cb8950f102, starting replication
2021/04/13 11:08:43 [INFO] raft: Added peer d0130513-a81e-5dda-0b84-625a0a08dfdb, starting replication
2021/04/13 11:08:43 [INFO] consul: cluster leadership acquired
2021/04/13 11:08:43 [INFO] consul: New leader elected: dev-consul-server-1
2021/04/13 11:08:43 [INFO] raft: pipelining replication to peer {Voter d0130513-a81e-5dda-0b84-625a0a08dfdb 10.242.209.68:8300}
2021/04/13 11:08:43 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' joined, marking health alive
2021/04/13 11:08:43 [WARN] raft: Rejecting vote request from 10.242.213.219:8300 since we have a leader: 10.242.214.76:8300
2021/04/13 11:08:45 [WARN] raft: AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300} rejected, sending older logs (next: 533)
2021/04/13 11:08:45 [INFO] raft: pipelining replication to peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}
2021/04/13 11:09:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:55300->10.242.213.219:8301: i/o timeout
2021/04/13 11:09:04 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:09:06 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:09:06 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/13 11:09:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:09:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:58102->10.242.213.207:8301: i/o timeout
2021/04/13 11:09:07 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:09:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:09:16 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:09:17 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500121971s
2021/04/13 11:09:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:09:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:58284->10.242.213.207:8301: i/o timeout
2021/04/13 11:09:20 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:09:22 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:09:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:09:23 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:09:23 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:09:23 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' failed, marking health critical
2021/04/13 11:09:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:58300->10.242.213.207:8301: i/o timeout
2021/04/13 11:09:23 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:09:24 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500220439s
2021/04/13 11:09:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:09:27 [ERR] memberlist: Push/Pull with dev-consul-7467bf7b46-8cwfv failed: read tcp 10.242.214.76:58234->10.242.213.207:8301: i/o timeout
2021/04/13 11:09:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:09:30 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:09:30 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/13 11:09:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:55628->10.242.213.219:8301: i/o timeout
2021/04/13 11:09:30 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:09:31 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500216703s
2021/04/13 11:09:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:09:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:09:33 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.99589602s
2021/04/13 11:09:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:55678->10.242.213.219:8301: i/o timeout
2021/04/13 11:09:33 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:09:35 [INFO] raft: aborting pipeline replication to peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}
2021/04/13 11:09:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:09:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:55692->10.242.213.219:8301: i/o timeout
2021/04/13 11:09:35 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:09:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:09:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:55706->10.242.213.219:8301: i/o timeout
2021/04/13 11:09:37 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:09:39 [INFO] raft: pipelining replication to peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}
2021/04/13 11:09:42 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:09:43 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:09:43 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' joined, marking health alive
2021/04/13 11:09:48 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:09:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:55904->10.242.213.219:8301: i/o timeout
2021/04/13 11:09:51 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:09:52 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:09:52 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500296602s
2021/04/13 11:09:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:09:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:58710->10.242.213.207:8301: i/o timeout
2021/04/13 11:09:53 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:09:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:09:57 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:09:57 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:09:57 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' failed, marking health critical
2021/04/13 11:09:57 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:09:57 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' joined, marking health alive
2021/04/13 11:09:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:56034->10.242.213.219:8301: i/o timeout
2021/04/13 11:09:58 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:10:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:10:08 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-8cwfv)
2021/04/13 11:10:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:10:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:56236->10.242.213.219:8301: i/o timeout
2021/04/13 11:10:12 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:10:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:10:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:59028->10.242.213.207:8301: i/o timeout
2021/04/13 11:10:14 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:10:16 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:10:16 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/13 11:10:16 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/13 11:10:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:59056->10.242.213.207:8301: i/o timeout
2021/04/13 11:10:16 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:10:18 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:10:18 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/13 11:10:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:10:19 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:10:19 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' failed, marking health critical
2021/04/13 11:10:22 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:10:22 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' joined, marking health alive
2021/04/13 11:10:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:56384->10.242.213.219:8301: i/o timeout
2021/04/13 11:10:24 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:10:55 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-8cwfv)
2021/04/13 11:11:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:11:26 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/13 11:11:26 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/13 11:11:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:59990->10.242.213.207:8301: i/o timeout
2021/04/13 11:11:26 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:11:27 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500313128s
2021/04/13 11:11:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:11:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:11:29 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.98975277s
2021/04/13 11:11:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57230->10.242.213.219:8301: i/o timeout
2021/04/13 11:11:29 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:11:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:11:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57288->10.242.213.219:8301: i/o timeout
2021/04/13 11:11:31 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:11:32 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 7.489626952s
2021/04/13 11:11:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:47848->10.242.213.219:8302: i/o timeout
2021/04/13 11:11:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:11:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:11:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:60100->10.242.213.207:8301: i/o timeout
2021/04/13 11:11:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:11:36 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:11:36 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-8cwfv)
2021/04/13 11:11:38 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:11:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:11:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57380->10.242.213.219:8301: i/o timeout
2021/04/13 11:11:39 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:11:40 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500340925s
2021/04/13 11:11:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:11:42 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.96293667s
2021/04/13 11:11:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:47992->10.242.213.219:8302: i/o timeout
2021/04/13 11:11:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:11:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:60188->10.242.213.207:8301: i/o timeout
2021/04/13 11:11:42 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:11:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:11:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:60238->10.242.213.207:8301: i/o timeout
2021/04/13 11:11:45 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:11:46 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:11:46 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/13 11:11:48 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:11:48 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:11:48 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' failed, marking health critical
2021/04/13 11:11:52 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:11:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:11:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:11:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:11:57 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/13 11:11:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57676->10.242.213.219:8301: i/o timeout
2021/04/13 11:11:58 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:11:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:00 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.50014629s
2021/04/13 11:12:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57728->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:01 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:02 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.979347983s
2021/04/13 11:12:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:04 [INFO] raft: aborting pipeline replication to peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}
2021/04/13 11:12:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57768->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:04 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57788->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:06 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:07 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500200599s
2021/04/13 11:12:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:07 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:12:07 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.213.219:42866
2021/04/13 11:12:08 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:46022->10.242.213.219:8300: i/o timeout
2021/04/13 11:12:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57802->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:08 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:09 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.992180261s
2021/04/13 11:12:10 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:12:10 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:12:10 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:12:10 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/13 11:12:10 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:12:10 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:12:10 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/13 11:12:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57862->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:11 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:12 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500285522s
2021/04/13 11:12:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57892->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:13 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:14 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:50194->10.242.213.219:8300: i/o timeout
2021/04/13 11:12:15 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.940558386s
2021/04/13 11:12:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57910->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:15 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:16 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:12:16 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' joined, marking health alive
2021/04/13 11:12:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57940->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:17 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:18 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:12:18 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:12:18 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:12:18 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/13 11:12:19 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:12:19 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:12:19 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/13 11:12:20 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:12:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/13 11:12:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:60768->10.242.213.207:8301: i/o timeout
2021/04/13 11:12:20 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:12:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:22 [INFO] raft: pipelining replication to peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}
2021/04/13 11:12:24 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:12:24 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/13 11:12:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:26 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.501464475s
2021/04/13 11:12:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:60854->10.242.213.207:8301: i/o timeout
2021/04/13 11:12:27 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:12:28 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/13 11:12:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:60916->10.242.213.207:8301: i/o timeout
2021/04/13 11:12:31 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:12:32 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500284404s
2021/04/13 11:12:32 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:12:32 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/13 11:12:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:34 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.995747831s
2021/04/13 11:12:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:60948->10.242.213.207:8301: i/o timeout
2021/04/13 11:12:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:12:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:37 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 7.483065923s
2021/04/13 11:12:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:58180->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:37 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:39 [INFO] raft: aborting pipeline replication to peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}
2021/04/13 11:12:40 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:50244->10.242.213.219:8300: i/o timeout
2021/04/13 11:12:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:58218->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:40 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:42 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500252887s
2021/04/13 11:12:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:48840->10.242.213.219:8302: i/o timeout
2021/04/13 11:12:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:12:43 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:12:43 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:12:43 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' failed, marking health critical
2021/04/13 11:12:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:32844->10.242.213.207:8301: i/o timeout
2021/04/13 11:12:43 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:12:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:58322->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:45 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:46 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:12:46 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:12:46 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:12:46 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/13 11:12:47 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.213.219:8301
2021/04/13 11:12:47 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500177187s
2021/04/13 11:12:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:58340->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:47 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:49 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.98065931s
2021/04/13 11:12:49 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:50660->10.242.213.219:8300: i/o timeout
2021/04/13 11:12:49 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:12:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:12:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:52 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 7.4310268s
2021/04/13 11:12:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:48898->10.242.213.219:8302: i/o timeout
2021/04/13 11:12:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:12:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:54 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/13 11:12:54 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:12:54 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:12:54 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/13 11:12:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:58504->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:12:56 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:12:56 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/13 11:12:56 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.503641164s
2021/04/13 11:12:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:12:57 [INFO] raft: pipelining replication to peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}
2021/04/13 11:12:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:58560->10.242.213.219:8301: i/o timeout
2021/04/13 11:12:59 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:13:00 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500196977s
2021/04/13 11:13:00 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:13:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:58586->10.242.213.219:8301: i/o timeout
2021/04/13 11:13:01 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:13:02 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.954604365s
2021/04/13 11:13:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:58652->10.242.213.219:8301: i/o timeout
2021/04/13 11:13:04 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:13:04 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 7.439617581s
2021/04/13 11:13:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:58678->10.242.213.219:8301: i/o timeout
2021/04/13 11:13:06 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:13:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:07 [INFO] raft: aborting pipeline replication to peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}
2021/04/13 11:13:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:49234->10.242.213.219:8302: i/o timeout
2021/04/13 11:13:07 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:13:07 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:50694->10.242.213.219:8300: i/o timeout
2021/04/13 11:13:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:58694->10.242.213.219:8301: i/o timeout
2021/04/13 11:13:08 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:13:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:10 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.504758304s
2021/04/13 11:13:10 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:13:10 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:13:10 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/13 11:13:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:58748->10.242.213.219:8301: i/o timeout
2021/04/13 11:13:10 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:13:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:12 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:13:12 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' joined, marking health alive
2021/04/13 11:13:12 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.983016883s
2021/04/13 11:13:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:33352->10.242.213.207:8301: i/o timeout
2021/04/13 11:13:13 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:13:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:33362->10.242.213.207:8301: i/o timeout
2021/04/13 11:13:14 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:13:15 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 7.44183156s
2021/04/13 11:13:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:33394->10.242.213.207:8301: i/o timeout
2021/04/13 11:13:16 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:13:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:17 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:51124->10.242.213.219:8300: i/o timeout
2021/04/13 11:13:17 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:13:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:33428->10.242.213.207:8301: i/o timeout
2021/04/13 11:13:18 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:13:18 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:51138->10.242.213.219:8300: i/o timeout
2021/04/13 11:13:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:19 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:13:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:49508->10.242.213.219:8302: i/o timeout
2021/04/13 11:13:22 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:13:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:33516->10.242.213.207:8301: i/o timeout
2021/04/13 11:13:22 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:13:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:33536->10.242.213.207:8301: i/o timeout
2021/04/13 11:13:24 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:13:25 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:13:25 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:13:25 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' failed, marking health critical
2021/04/13 11:13:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:27 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:51290->10.242.213.219:8300: i/o timeout
2021/04/13 11:13:27 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:13:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:29 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:51324->10.242.213.219:8300: i/o timeout
2021/04/13 11:13:30 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:13:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:49686->10.242.213.219:8302: i/o timeout
2021/04/13 11:13:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:13:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:34 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:13:34 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.213.219:50004
2021/04/13 11:13:34 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:13:34 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/13 11:13:34 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:13:34 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.213.207:41190
2021/04/13 11:13:36 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:13:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:37 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:51462->10.242.213.219:8300: i/o timeout
2021/04/13 11:13:37 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:13:37 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/13 11:13:37 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:13:37 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:13:37 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.213.219:8302
2021/04/13 11:13:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:40 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:51526->10.242.213.219:8300: i/o timeout
2021/04/13 11:13:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:41 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:13:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:49740->10.242.213.219:8302: i/o timeout
2021/04/13 11:13:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:13:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:47 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:51620->10.242.213.219:8300: i/o timeout
2021/04/13 11:13:47 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:13:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:51 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:51688->10.242.213.219:8300: i/o timeout
2021/04/13 11:13:52 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:13:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:54 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:13:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:13:57 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:51754->10.242.213.219:8300: i/o timeout
2021/04/13 11:13:57 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:13:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:02 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:51834->10.242.213.219:8300: i/o timeout
2021/04/13 11:14:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:03 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:14:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:06 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:14:06 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:14:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:07 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:14:07 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:51928->10.242.213.219:8300: i/o timeout
2021/04/13 11:14:08 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:14:09 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500282874s
2021/04/13 11:14:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:11 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.95356236s
2021/04/13 11:14:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:50254->10.242.213.219:8302: i/o timeout
2021/04/13 11:14:12 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:14:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:14 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 7.432022769s
2021/04/13 11:14:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:15 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:14:15 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/13 11:14:17 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:52016->10.242.213.219:8300: i/o timeout
2021/04/13 11:14:18 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:52058->10.242.213.219:8300: i/o timeout
2021/04/13 11:14:18 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:14:18 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:14:18 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:14:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:50444->10.242.213.219:8302: i/o timeout
2021/04/13 11:14:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:14:28 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:52196->10.242.213.219:8300: i/o timeout
2021/04/13 11:14:28 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:52198->10.242.213.219:8300: i/o timeout
2021/04/13 11:14:29 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:14:29 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:14:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:34 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.213.219:8301
2021/04/13 11:14:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:39 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:52332->10.242.213.219:8300: i/o timeout
2021/04/13 11:14:39 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:52336->10.242.213.219:8300: i/o timeout
2021/04/13 11:14:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:40 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:14:40 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:14:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:42 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/13 11:14:42 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:14:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:50580->10.242.213.219:8302: i/o timeout
2021/04/13 11:14:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:14:42 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:14:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:50 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:52488->10.242.213.219:8300: i/o timeout
2021/04/13 11:14:50 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:52500->10.242.213.219:8300: i/o timeout
2021/04/13 11:14:50 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:14:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:52 [ERR] memberlist: failed to receive: read tcp 10.242.214.76:8301->10.242.213.207:32824: i/o timeout from=10.242.213.207:32824
2021/04/13 11:14:52 [ERR] memberlist: Failed to send error: write tcp 10.242.214.76:8301->10.242.213.207:32824: i/o timeout from=10.242.213.207:32824
2021/04/13 11:14:52 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:14:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:14:57 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:14:57 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/13 11:15:00 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:15:00 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:52632->10.242.213.219:8300: i/o timeout
2021/04/13 11:15:01 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:15:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:02 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:52678->10.242.213.219:8300: i/o timeout
2021/04/13 11:15:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:06 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.213.219:8302
2021/04/13 11:15:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:07 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:15:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:11 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:52824->10.242.213.219:8300: i/o timeout
2021/04/13 11:15:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:12 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:15:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:14 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:15:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:17 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:52902->10.242.213.219:8300: i/o timeout
2021/04/13 11:15:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:22 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:52976->10.242.213.219:8300: i/o timeout
2021/04/13 11:15:23 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:15:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:28 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:15:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:33 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:53114->10.242.213.219:8300: i/o timeout
2021/04/13 11:15:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:34 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:15:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:38 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:53162->10.242.213.219:8300: i/o timeout
2021/04/13 11:15:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:39 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:15:39 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/13 11:15:42 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:15:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:44 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:53256->10.242.213.219:8300: i/o timeout
2021/04/13 11:15:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:46 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:15:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:48 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:15:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:54 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:15:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:56 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:53404->10.242.213.219:8300: i/o timeout
2021/04/13 11:15:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:58 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:53428->10.242.213.219:8300: i/o timeout
2021/04/13 11:15:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:15:59 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:16:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:08 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:16:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:09 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:53618->10.242.213.219:8300: i/o timeout
2021/04/13 11:16:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:15 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:16:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:18 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:53746->10.242.213.219:8300: i/o timeout
2021/04/13 11:16:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:21 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:16:21 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/13 11:16:23 [WARN] raft: Rejecting vote request from 10.242.213.219:8300 since we have a leader: 10.242.214.76:8300
2021/04/13 11:16:24 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:16:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:25 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:16:25 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:16:25 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/13 11:16:25 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:53838->10.242.213.219:8300: i/o timeout
2021/04/13 11:16:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:33340->10.242.213.219:8301: i/o timeout
2021/04/13 11:16:27 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:16:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:33364->10.242.213.219:8301: i/o timeout
2021/04/13 11:16:29 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:16:31 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (0 peer confirmations)
2021/04/13 11:16:31 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:16:31 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:16:31 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/13 11:16:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:33408->10.242.213.219:8301: i/o timeout
2021/04/13 11:16:31 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:16:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:34 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.213.219:8301
2021/04/13 11:16:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:36 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:16:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:39 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:54012->10.242.213.219:8300: i/o timeout
2021/04/13 11:16:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:46 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:54124->10.242.213.219:8300: i/o timeout
2021/04/13 11:16:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:49 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:16:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:53 [ERR] memberlist: failed to receive: read tcp 10.242.214.76:8302->10.242.213.219:54082: i/o timeout from=10.242.213.219:54082
2021/04/13 11:16:53 [ERR] memberlist: Failed to send error: write tcp 10.242.214.76:8302->10.242.213.219:54082: i/o timeout from=10.242.213.219:54082
2021/04/13 11:16:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:16:57 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:16:59 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:54282->10.242.213.219:8300: i/o timeout
2021/04/13 11:16:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:03 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:17:03 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/13 11:17:06 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:17:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:07 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:54444->10.242.213.219:8300: i/o timeout
2021/04/13 11:17:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:09 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:17:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:14 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:17:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:18 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:17:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:19 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:54596->10.242.213.219:8300: i/o timeout
2021/04/13 11:17:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:28 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:54718->10.242.213.219:8300: i/o timeout
2021/04/13 11:17:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:29 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:17:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:39 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:17:39 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:54862->10.242.213.219:8300: i/o timeout
2021/04/13 11:17:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:45 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:17:45 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/13 11:17:48 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:17:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:49 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:54994->10.242.213.219:8300: i/o timeout
2021/04/13 11:17:49 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:17:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:54 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:17:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:17:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:00 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:55132->10.242.213.219:8300: i/o timeout
2021/04/13 11:18:00 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:18:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:06 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.213.219:8302
2021/04/13 11:18:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:10 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:18:10 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:55326->10.242.213.219:8300: i/o timeout
2021/04/13 11:18:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:20 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:55450->10.242.213.219:8300: i/o timeout
2021/04/13 11:18:21 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:18:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:27 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:18:27 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/13 11:18:30 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:18:30 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/13 11:18:30 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:18:31 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:55598->10.242.213.219:8300: i/o timeout
2021/04/13 11:18:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:34 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:18:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:40 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:55718->10.242.213.219:8300: i/o timeout
2021/04/13 11:18:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:42 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:18:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:50 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:18:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:52 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:55890->10.242.213.219:8300: i/o timeout
2021/04/13 11:18:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:18:59 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:18:59 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.213.219:52012
2021/04/13 11:18:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:00 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:55992->10.242.213.219:8300: i/o timeout
2021/04/13 11:19:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:02 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:19:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:09 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:19:09 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/13 11:19:10 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:19:12 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:19:12 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:56210->10.242.213.219:8300: i/o timeout
2021/04/13 11:19:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:14 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:19:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:16 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:19:16 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:19:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:20 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:56304->10.242.213.219:8300: i/o timeout
2021/04/13 11:19:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:23 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:19:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:31 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:19:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:54796->10.242.213.219:8302: i/o timeout
2021/04/13 11:19:32 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:19:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:33 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:56472->10.242.213.219:8300: i/o timeout
2021/04/13 11:19:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:41 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:56666->10.242.213.219:8300: i/o timeout
2021/04/13 11:19:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:41 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-pmks5 10.242.110.227
2021/04/13 11:19:41 [INFO] consul: member 'dev-consul-7467bf7b46-pmks5' joined, marking health alive
2021/04/13 11:19:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:54990->10.242.213.219:8302: i/o timeout
2021/04/13 11:19:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:19:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:44 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:19:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:51 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:19:51 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/13 11:19:51 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:19:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:55148->10.242.213.219:8302: i/o timeout
2021/04/13 11:19:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:19:54 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:19:54 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:56900->10.242.213.219:8300: i/o timeout
2021/04/13 11:19:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:59 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/13 11:19:59 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:19:59 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:19:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:19:59 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/13 11:20:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:01 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:57016->10.242.213.219:8300: i/o timeout
2021/04/13 11:20:02 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured
2021/04/13 11:20:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:05 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:20:06 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.213.219:8302
2021/04/13 11:20:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:09 [ERR] memberlist: failed to receive: read tcp 10.242.214.76:8301->10.242.213.219:39058: i/o timeout from=10.242.213.219:39058
2021/04/13 11:20:09 [ERR] memberlist: Failed to send error: write tcp 10.242.214.76:8301->10.242.213.219:39058: i/o timeout from=10.242.213.219:39058
2021/04/13 11:20:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:11 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:20:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:14 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:20:14 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:20:14 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:20:14 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:20:14 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/13 11:20:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:15 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:57254->10.242.213.219:8300: i/o timeout
2021/04/13 11:20:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:36742->10.242.213.219:8301: i/o timeout
2021/04/13 11:20:15 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:20:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:19 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:20:19 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:20:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:36790->10.242.213.219:8301: i/o timeout
2021/04/13 11:20:19 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:20:19 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:20:19 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/13 11:20:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:21 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:57356->10.242.213.219:8300: i/o timeout
2021/04/13 11:20:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:24 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.213.219:8301
2021/04/13 11:20:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:26 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:20:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:55694->10.242.213.219:8302: i/o timeout
2021/04/13 11:20:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:20:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:31 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:20:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:33 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:20:33 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/13 11:20:36 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:57556->10.242.213.219:8300: i/o timeout
2021/04/13 11:20:36 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:20:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:41 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:57658->10.242.213.219:8300: i/o timeout
2021/04/13 11:20:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:55854->10.242.213.219:8302: i/o timeout
2021/04/13 11:20:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:20:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:47 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:20:49 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/13 11:20:49 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:20:49 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:20:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:52 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:20:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:56174->10.242.213.219:8302: i/o timeout
2021/04/13 11:20:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:20:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:53 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:20:54 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:20:54 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.213.219:52218
2021/04/13 11:20:55 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2)
2021/04/13 11:20:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:55 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.508562139s
2021/04/13 11:20:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:20:58 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.964565925s
2021/04/13 11:20:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:00 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 7.395471107s
2021/04/13 11:21:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:02 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:58064->10.242.213.219:8300: i/o timeout
2021/04/13 11:21:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:03 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:57936->10.242.213.219:8300: i/o timeout
2021/04/13 11:21:04 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:21:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:12 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:21:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:14 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.213.219:8302
2021/04/13 11:21:14 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:58390->10.242.213.219:8300: i/o timeout
2021/04/13 11:21:15 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:21:15 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:21:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:15 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/13 11:21:18 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:21:19 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1)
2021/04/13 11:21:19 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:21:19 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:21:19 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:21:19 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:21:19 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/13 11:21:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:21 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500300712s
2021/04/13 11:21:22 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:21:22 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/13 11:21:22 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:58522->10.242.213.219:8300: i/o timeout
2021/04/13 11:21:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:23 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.957847027s
2021/04/13 11:21:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:38070->10.242.213.219:8301: i/o timeout
2021/04/13 11:21:24 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:21:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:25 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:21:25 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:21:25 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:21:25 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/13 11:21:26 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:21:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:27 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.502154662s
2021/04/13 11:21:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:30 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.930814005s
2021/04/13 11:21:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:32 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:21:32 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 7.411727551s
2021/04/13 11:21:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:34 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.213.219:8301
2021/04/13 11:21:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:36 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:58584->10.242.213.219:8300: i/o timeout
2021/04/13 11:21:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:37 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:21:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:40 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:21:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:41 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.501995115s
2021/04/13 11:21:42 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:58862->10.242.213.219:8300: i/o timeout
2021/04/13 11:21:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57162->10.242.213.219:8302: i/o timeout
2021/04/13 11:21:42 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:21:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:44 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.939875484s
2021/04/13 11:21:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:46 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 7.424733307s
2021/04/13 11:21:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:50 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:58956->10.242.213.219:8300: i/o timeout
2021/04/13 11:21:50 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:21:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57232->10.242.213.219:8302: i/o timeout
2021/04/13 11:21:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:21:52 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:21:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:21:57 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:21:57 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/13 11:22:00 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:22:00 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:59170->10.242.213.219:8300: i/o timeout
2021/04/13 11:22:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:01 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:22:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:57524->10.242.213.219:8302: i/o timeout
2021/04/13 11:22:02 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:22:02 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:59218->10.242.213.219:8300: i/o timeout
2021/04/13 11:22:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:04 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/13 11:22:04 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:22:04 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:22:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:08 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:22:08 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:22:08 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/13 11:22:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:38850->10.242.213.219:8301: i/o timeout
2021/04/13 11:22:10 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:22:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:11 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:59396->10.242.213.219:8300: i/o timeout
2021/04/13 11:22:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:38918->10.242.213.219:8301: i/o timeout
2021/04/13 11:22:13 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:22:14 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:22:14 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:22:14 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:22:14 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/13 11:22:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:22 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:59564->10.242.213.219:8300: i/o timeout
2021/04/13 11:22:23 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:59568->10.242.213.219:8300: i/o timeout
2021/04/13 11:22:23 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:22:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:28 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:22:28 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:22:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:33 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:59716->10.242.213.219:8300: i/o timeout
2021/04/13 11:22:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:33 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:22:34 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:22:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:58074->10.242.213.219:8302: i/o timeout
2021/04/13 11:22:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:22:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:39 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:22:39 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/13 11:22:42 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:22:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:43 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:59904->10.242.213.219:8300: i/o timeout
2021/04/13 11:22:44 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:59910->10.242.213.219:8300: i/o timeout
2021/04/13 11:22:44 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-8cwfv 10.242.213.207:8301
2021/04/13 11:22:44 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:22:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:45 [ERR] memberlist: failed to receive: read tcp 10.242.214.76:8301->10.242.213.207:35924: i/o timeout from=10.242.213.207:35924
2021/04/13 11:22:45 [ERR] memberlist: Failed to send error: write tcp 10.242.214.76:8301->10.242.213.207:35924: i/o timeout from=10.242.213.207:35924
2021/04/13 11:22:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:48 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:22:48 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.213.219:54230
2021/04/13 11:22:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:58320->10.242.213.219:8302: i/o timeout
2021/04/13 11:22:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:22:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:53 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:22:54 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:60072->10.242.213.219:8300: i/o timeout
2021/04/13 11:22:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:55 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:22:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:22:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:58520->10.242.213.219:8302: i/o timeout
2021/04/13 11:23:02 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:23:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:03 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:60224->10.242.213.219:8300: i/o timeout
2021/04/13 11:23:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:06 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:60246->10.242.213.219:8300: i/o timeout
2021/04/13 11:23:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:07 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:23:07 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/13 11:23:07 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:23:07 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:23:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:58594->10.242.213.219:8302: i/o timeout
2021/04/13 11:23:12 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:23:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:13 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:23:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:17 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:60474->10.242.213.219:8300: i/o timeout
2021/04/13 11:23:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:19 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:23:19 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:23:19 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' joined, marking health alive
2021/04/13 11:23:21 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:21 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:23:21 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/13 11:23:23 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:60584->10.242.213.219:8300: i/o timeout
2021/04/13 11:23:24 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:23:24 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.213.219:8301
2021/04/13 11:23:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:42990->10.242.213.207:8301: i/o timeout
2021/04/13 11:23:25 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:23:26 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:23:26 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:23:26 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' failed, marking health critical
2021/04/13 11:23:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:29 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:60658->10.242.213.219:8300: i/o timeout
2021/04/13 11:23:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:32 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:23:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:34 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:23:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:42 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: read tcp 10.242.214.76:32772->10.242.213.219:8300: i/o timeout
2021/04/13 11:23:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:44 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:32784->10.242.213.219:8300: i/o timeout
2021/04/13 11:23:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:48 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:23:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:54 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:23:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:57 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:23:59 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:23:59 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' joined, marking health alive
2021/04/13 11:23:59 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:23:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:24:00 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:24:01 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:24:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:24:01 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:24:02 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:24:03 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:24:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:24:03 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/13 11:24:03 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: EOF
2021/04/13 11:24:03 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-8cwfv but other probes failed, network may be misconfigured
2021/04/13 11:24:04 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:24:04 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: read tcp 10.242.214.76:33096->10.242.213.219:8300: i/o timeout
2021/04/13 11:24:04 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.213.219:8301
2021/04/13 11:24:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:43662->10.242.213.207:8301: i/o timeout
2021/04/13 11:24:04 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:24:05 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:24:05 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:24:05 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' failed, marking health critical
2021/04/13 11:24:05 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:24:05 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' joined, marking health alive
2021/04/13 11:24:05 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500268277s
2021/04/13 11:24:06 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:24:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:24:08 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.932347101s
2021/04/13 11:24:09 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:24:09 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:24:09 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:24:09 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/13 11:24:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:43722->10.242.213.207:8301: i/o timeout
2021/04/13 11:24:09 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:24:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:24:11 [INFO] memberlist: Marking dev-consul-7467bf7b46-8cwfv as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:24:11 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:24:11 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' failed, marking health critical
2021/04/13 11:24:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:41014->10.242.213.219:8301: i/o timeout
2021/04/13 11:24:12 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:24:12 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-8cwfv)
2021/04/13 11:24:12 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:24:12 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:24:13 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:24:13 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' joined, marking health alive
2021/04/13 11:24:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:41038->10.242.213.219:8301: i/o timeout
2021/04/13 11:24:15 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:24:16 [INFO] raft: pipelining replication to peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}
2021/04/13 11:24:18 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:24:20 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:24:20 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/13 11:24:28 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:24:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:24:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:44060->10.242.213.207:8301: i/o timeout
2021/04/13 11:24:30 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:24:40 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:24:42 [WARN] memberlist: Was able to connect to dev-consul-server-2 but other probes failed, network may be misconfigured
2021/04/13 11:24:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:41520->10.242.213.219:8301: i/o timeout
2021/04/13 11:24:43 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:24:44 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:24:45 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500165492s
2021/04/13 11:24:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:24:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:24:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.76:44318->10.242.213.207:8301: i/o timeout
2021/04/13 11:24:47 [INFO] memberlist: Suspect dev-consul-7467bf7b46-8cwfv has failed, no acks received
2021/04/13 11:24:47 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations)
2021/04/13 11:24:47 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:24:47 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:24:47 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/13 11:24:47 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.997846899s
2021/04/13 11:24:48 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.219
2021/04/13 11:24:48 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:24:48 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/13 11:24:48 [WARN] raft: Rejecting vote request from 10.242.213.219:8300 since we have a leader: 10.242.214.76:8300
2021/04/13 11:24:48 [ERR] raft: peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300} has newer term, stopping replication
2021/04/13 11:24:48 [INFO] raft: aborting pipeline replication to peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}
2021/04/13 11:24:48 [INFO] raft: Node at 10.242.214.76:8300 [Follower] entering Follower state (Leader: "")
2021/04/13 11:24:48 [INFO] raft: aborting pipeline replication to peer {Voter d0130513-a81e-5dda-0b84-625a0a08dfdb 10.242.209.68:8300}
2021/04/13 11:24:48 [INFO] consul: cluster leadership lost
2021/04/13 11:24:51 [WARN] consul.coordinate: Batch update failed: node is not the leader
2021/04/13 11:24:57 [WARN] raft: Rejecting vote request from 10.242.213.219:8300 since our last index is greater (1018, 1015)
2021/04/13 11:24:57 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/13 11:24:57 [INFO] raft: Node at 10.242.214.76:8300 [Candidate] entering Candidate state in term 11
2021/04/13 11:24:57 [INFO] raft: Election won. Tally: 2
2021/04/13 11:24:57 [INFO] raft: Node at 10.242.214.76:8300 [Leader] entering Leader state
2021/04/13 11:24:57 [INFO] raft: Added peer dfed536e-3d79-cd25-9127-f2cb8950f102, starting replication
2021/04/13 11:24:57 [INFO] raft: Added peer d0130513-a81e-5dda-0b84-625a0a08dfdb, starting replication
2021/04/13 11:24:57 [INFO] consul: cluster leadership acquired
2021/04/13 11:24:57 [INFO] consul: New leader elected: dev-consul-server-1
2021/04/13 11:24:57 [INFO] raft: pipelining replication to peer {Voter d0130513-a81e-5dda-0b84-625a0a08dfdb 10.242.209.68:8300}
2021/04/13 11:24:59 [WARN] raft: AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300} rejected, sending older logs (next: 1016)
2021/04/13 11:24:59 [INFO] raft: pipelining replication to peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}
2021/04/13 11:25:06 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:25:06 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/13 11:25:18 [INFO] raft: aborting pipeline replication to peer {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}
2021/04/13 11:25:18 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: EOF
2021/04/13 11:25:18 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: dial tcp ->10.242.213.219:8300: getsockopt: connection refused
2021/04/13 11:25:18 [INFO] serf: EventMemberLeave: dev-consul-7467bf7b46-8cwfv 10.242.213.207
2021/04/13 11:25:18 [INFO] consul: member 'dev-consul-7467bf7b46-8cwfv' left, deregistering
2021/04/13 11:25:18 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: dial tcp ->10.242.213.219:8300: getsockopt: connection refused
2021/04/13 11:25:18 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: dial tcp ->10.242.213.219:8300: getsockopt: connection refused
2021/04/13 11:25:18 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: dial tcp ->10.242.213.219:8300: getsockopt: connection refused
2021/04/13 11:25:18 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: dial tcp ->10.242.213.219:8300: getsockopt: connection refused
2021/04/13 11:25:18 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: dial tcp ->10.242.213.219:8300: getsockopt: connection refused
2021/04/13 11:25:18 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: dial tcp ->10.242.213.219:8300: getsockopt: connection refused
2021/04/13 11:25:19 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: dial tcp ->10.242.213.219:8300: getsockopt: connection refused
2021/04/13 11:25:19 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: dial tcp ->10.242.213.219:8300: getsockopt: connection refused
2021/04/13 11:25:19 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error making call: stream closed
2021/04/13 11:25:19 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: dial tcp ->10.242.213.219:8300: getsockopt: connection refused
2021/04/13 11:25:20 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: dial tcp ->10.242.213.219:8300: getsockopt: connection refused
2021/04/13 11:25:20 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:25:20 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/13 11:25:20 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: dial tcp ->10.242.213.219:8300: getsockopt: connection refused
2021/04/13 11:25:20 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 2.500314472s
2021/04/13 11:25:22 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:25:23 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 4.978016948s
2021/04/13 11:25:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:25:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:25:25 [WARN] raft: Failed to contact dfed536e-3d79-cd25-9127-f2cb8950f102 in 7.436796365s
2021/04/13 11:25:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:25:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:25:31 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:25:31 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (1 peer confirmations)
2021/04/13 11:25:31 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.213.219
2021/04/13 11:25:31 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.213.219:8300) (DC: dc1)
2021/04/13 11:25:31 [INFO] consul: member 'dev-consul-server-2' failed, marking health critical
2021/04/13 11:25:31 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:25:31 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:25:31 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:25:31 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:25:32 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received
2021/04/13 11:25:34 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:25:34 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:25:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:25:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:25:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:25:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:25:41 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:25:42 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:25:43 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:25:43 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:25:44 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:25:46 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:25:46 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/13 11:25:46 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/13 11:25:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:25:49 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:25:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:25:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:25:52 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:25:53 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:25:53 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:25:55 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:25:55 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:25:55 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:25:58 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:25:58 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.213.219:8302 Theirs: 10.242.213.195:8302
2021/04/13 11:25:58 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.213.219:8302 and 10.242.213.195:8302 are claiming
2021/04/13 11:25:58 [ERR] memberlist: Conflicting address for dev-consul-server-2. Mine: 10.242.213.219:8301 Theirs: 10.242.213.195:8301
2021/04/13 11:25:58 [WARN] serf: Name conflict for 'dev-consul-server-2' both 10.242.213.219:8301 and 10.242.213.195:8301 are claiming
2021/04/13 11:25:59 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:25:59 [ERR] memberlist: Conflicting address for dev-consul-server-2. Mine: 10.242.213.219:8301 Theirs: 10.242.213.195:8301
2021/04/13 11:25:59 [WARN] serf: Name conflict for 'dev-consul-server-2' both 10.242.213.219:8301 and 10.242.213.195:8301 are claiming
2021/04/13 11:25:59 [ERR] memberlist: Conflicting address for dev-consul-server-2. Mine: 10.242.213.219:8301 Theirs: 10.242.213.195:8301
2021/04/13 11:25:59 [WARN] serf: Name conflict for 'dev-consul-server-2' both 10.242.213.219:8301 and 10.242.213.195:8301 are claiming
2021/04/13 11:25:59 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.213.219:8302 Theirs: 10.242.213.195:8302
2021/04/13 11:25:59 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.213.219:8302 and 10.242.213.195:8302 are claiming
2021/04/13 11:25:59 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.213.219:8302 Theirs: 10.242.213.195:8302
2021/04/13 11:25:59 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.213.219:8302 and 10.242.213.195:8302 are claiming
2021/04/13 11:25:59 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:25:59 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.213.219:8302 Theirs: 10.242.213.195:8302
2021/04/13 11:25:59 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.213.219:8302 and 10.242.213.195:8302 are claiming
2021/04/13 11:25:59 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.213.219:8302 Theirs: 10.242.213.195:8302
2021/04/13 11:25:59 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.213.219:8302 and 10.242.213.195:8302 are claiming
2021/04/13 11:26:01 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:26:03 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:26:03 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:26:05 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:26:05 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:26:07 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:26:07 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:26:09 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:26:10 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:26:11 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:26:13 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:26:15 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:26:15 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:26:17 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:26:17 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:26:19 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:26:19 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:26:19 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:26:22 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:26:23 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:26:24 [ERR] memberlist: Push/Pull with dev-consul-server-2.dc1 failed: dial tcp 10.242.213.219:8302: i/o timeout
2021/04/13 11:26:25 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:26:25 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/13 11:26:25 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.213.219
2021/04/13 11:26:25 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/13 11:26:27 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:26:27 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:26:29 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:26:29 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:26:30 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:26:31 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:26:32 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:26:33 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:26:35 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:26:35 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received
2021/04/13 11:26:37 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:26:39 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:26:39 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:26:40 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:26:41 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:26:41 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:26:44 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/13 11:26:45 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:26:46 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:26:47 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:26:49 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.213.219:8302
2021/04/13 11:26:49 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:26:49 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:26:51 [WARN] consul: error getting server health from "dev-consul-server-2": last request still outstanding
2021/04/13 11:26:51 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.213.195
2021/04/13 11:26:51 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.213.195:8300) (DC: dc1)
2021/04/13 11:26:51 [INFO] raft: Updating configuration with AddNonvoter (c5a8c987-4cb7-80ed-728c-e6cec17afe6f, 10.242.213.195:8300) to [{Suffrage:Voter ID:f589d1e2-bc2b-3068-9a75-b76c06ae0d2b Address:10.242.214.76:8300} {Suffrage:Voter ID:dfed536e-3d79-cd25-9127-f2cb8950f102 Address:10.242.213.219:8300} {Suffrage:Voter ID:d0130513-a81e-5dda-0b84-625a0a08dfdb Address:10.242.209.68:8300} {Suffrage:Nonvoter ID:c5a8c987-4cb7-80ed-728c-e6cec17afe6f Address:10.242.213.195:8300}]
2021/04/13 11:26:51 [INFO] raft: Added peer c5a8c987-4cb7-80ed-728c-e6cec17afe6f, starting replication
2021/04/13 11:26:51 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/04/13 11:26:51 [WARN] raft: AppendEntries to {Nonvoter c5a8c987-4cb7-80ed-728c-e6cec17afe6f 10.242.213.195:8300} rejected, sending older logs (next: 1)
2021/04/13 11:26:52 [INFO] raft: pipelining replication to peer {Nonvoter c5a8c987-4cb7-80ed-728c-e6cec17afe6f 10.242.213.195:8300}
2021/04/13 11:26:53 [WARN] consul: error getting server health from "dev-consul-server-2": rpc error getting client: failed to get conn: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:26:56 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:26:59 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:27:07 [INFO] autopilot: Promoting Server (ID: "c5a8c987-4cb7-80ed-728c-e6cec17afe6f" Address: "10.242.213.195:8300") to voter
2021/04/13 11:27:07 [INFO] raft: Updating configuration with AddStaging (c5a8c987-4cb7-80ed-728c-e6cec17afe6f, 10.242.213.195:8300) to [{Suffrage:Voter ID:f589d1e2-bc2b-3068-9a75-b76c06ae0d2b Address:10.242.214.76:8300} {Suffrage:Voter ID:dfed536e-3d79-cd25-9127-f2cb8950f102 Address:10.242.213.219:8300} {Suffrage:Voter ID:d0130513-a81e-5dda-0b84-625a0a08dfdb Address:10.242.209.68:8300} {Suffrage:Voter ID:c5a8c987-4cb7-80ed-728c-e6cec17afe6f Address:10.242.213.195:8300}]
2021/04/13 11:27:07 [INFO] autopilot: Attempting removal of stale Server (ID: "dfed536e-3d79-cd25-9127-f2cb8950f102" Address: "10.242.213.219:8300")
2021/04/13 11:27:07 [INFO] raft: Updating configuration with RemoveServer (dfed536e-3d79-cd25-9127-f2cb8950f102, ) to [{Suffrage:Voter ID:f589d1e2-bc2b-3068-9a75-b76c06ae0d2b Address:10.242.214.76:8300} {Suffrage:Voter ID:d0130513-a81e-5dda-0b84-625a0a08dfdb Address:10.242.209.68:8300} {Suffrage:Voter ID:c5a8c987-4cb7-80ed-728c-e6cec17afe6f Address:10.242.213.195:8300}]
2021/04/13 11:27:07 [INFO] raft: Removed peer dfed536e-3d79-cd25-9127-f2cb8950f102, stopping replication after 1083
2021/04/13 11:27:07 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:27:10 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:27:17 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:27:20 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:27:28 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:27:30 [WARN] Unable to get address for server id dfed536e-3d79-cd25-9127-f2cb8950f102, using fallback address 10.242.213.219:8300: Could not find address for server id dfed536e-3d79-cd25-9127-f2cb8950f102
2021/04/13 11:27:38 [ERR] raft: Failed to heartbeat to 10.242.213.219:8300: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:27:40 [ERR] raft: Failed to AppendEntries to {Voter dfed536e-3d79-cd25-9127-f2cb8950f102 10.242.213.219:8300}: dial tcp ->10.242.213.219:8300: i/o timeout
2021/04/13 11:27:58 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.213.195
2021/04/13 11:27:58 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"