By type
+ CONSUL_BIND=
+ [ -n ]
+ CONSUL_CLIENT=
+ [ -n ]
+ CONSUL_DATA_DIR=/consul/data
+ CONSUL_CONFIG_DIR=/consul/config
+ [ -n ]
+ [ a = - ]
+ [ agent = agent ]
+ shift
+ set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
+ [ consul = consul ]
+ stat -c %u /consul/data
+ id -u consul
+ [ 100 != 100 ]
+ stat -c %u /consul/config
+ id -u consul
+ [ 100 != 100 ]
+ [ ! -z ]
+ exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
bootstrap_expect > 0: expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
Version: 'v1.0.6'
Node ID: '030af47e-b80c-8935-3357-782656bd05e5'
Node name: 'dev-consul-server-2'
Datacenter: 'dc1' (Segment: '')
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.242.139.204 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2021/05/07 10:43:54 [INFO] raft: Initial configuration (index=0): []
2021/05/07 10:43:54 [INFO] raft: Node at 10.242.139.204:8300 [Follower] entering Follower state (Leader: "")
2021/05/07 10:43:54 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.139.204
2021/05/07 10:43:54 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.139.204
2021/05/07 10:43:54 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/05/07 10:43:54 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2021/05/07 10:43:54 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.139.204:8300) (DC: dc1)
2021/05/07 10:43:54 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2021/05/07 10:43:54 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
2021/05/07 10:43:54 [INFO] agent: started state syncer
2021/05/07 10:43:54 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer
2021/05/07 10:43:54 [INFO] agent: Joining LAN cluster...
2021/05/07 10:43:54 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/05/07 10:43:54 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.113.198
2021/05/07 10:43:54 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 10:43:54 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 10:43:54 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.113.198:8300) (DC: dc1)
2021/05/07 10:43:54 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.242.139.204:8300,10.242.113.198:8300,10.242.41.146:8300
2021/05/07 10:43:54 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 10:43:54 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 10:43:54 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.113.198
2021/05/07 10:43:54 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 10:43:54 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/05/07 10:43:54 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/05/07 10:43:54 [INFO] agent: (LAN) joined: 2 Err:
2021/05/07 10:43:54 [INFO] agent: Join LAN completed. Synced with 2 initial agents
2021/05/07 10:44:00 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/05/07 10:44:00 [INFO] raft: Node at 10.242.139.204:8300 [Candidate] entering Candidate state in term 2
2021/05/07 10:44:00 [INFO] raft: Election won. Tally: 2
2021/05/07 10:44:00 [INFO] raft: Node at 10.242.139.204:8300 [Leader] entering Leader state
2021/05/07 10:44:00 [INFO] raft: Added peer dff60f1e-22b9-7f6a-153b-e2e4aea758bb, starting replication
2021/05/07 10:44:00 [INFO] raft: Added peer e90a9b84-7b02-fc1f-15b4-2419fe66fc38, starting replication
2021/05/07 10:44:00 [INFO] consul: cluster leadership acquired
2021/05/07 10:44:00 [INFO] consul: New leader elected: dev-consul-server-2
2021/05/07 10:44:00 [WARN] raft: AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300} rejected, sending older logs (next: 1)
2021/05/07 10:44:00 [INFO] raft: pipelining replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 10:44:00 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive
2021/05/07 10:44:00 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/05/07 10:44:00 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 10:44:00 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 10:44:00 [INFO] agent: Synced node info
2021/05/07 10:44:02 [WARN] raft: Failed to contact dff60f1e-22b9-7f6a-153b-e2e4aea758bb in 2.501647836s
2021/05/07 10:44:03 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:44:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:44:05 [INFO] raft: pipelining replication to peer {Voter dff60f1e-22b9-7f6a-153b-e2e4aea758bb 10.242.113.198:8300}
2021/05/07 10:44:07 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:44:07 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=ceafccc7-a797-53bf-c595-db8814e117aa&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2021/05/07 10:44:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:44:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:44:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:44:15 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:44:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:44:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:44:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:44:23 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:44:23 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 10:44:51 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:44:51 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 10:44:55 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:44:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:44:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:45:01 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:45:03 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:45:55 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:46:05 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:46:09 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:46:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:46:25 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:46:41 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:46:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:48:17 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:48:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:48:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:48:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:48:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:48:47 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:48:47 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 10:49:17 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:49:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:49:21 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:49:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:49:25 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:49:27 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:49:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:49:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:49:49 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:49:51 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:49:51 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 10:50:01 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:50:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:50:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:50:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:50:23 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:51:01 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:51:01 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 10:51:03 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:51:03 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 10:51:13 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:51:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:52:27 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:52:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:54:17 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:57:23 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:59:13 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:59:23 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 10:59:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 10:59:31 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 11:05:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:33432->10.242.41.137:8301: i/o timeout
2021/05/07 11:05:58 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:06:01 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:06:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:33474->10.242.41.137:8301: i/o timeout
2021/05/07 11:06:01 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:06:31 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:06:31 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/07 11:06:31 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 11:06:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:33972->10.242.41.137:8301: i/o timeout
2021/05/07 11:06:33 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:07:17 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:07:27 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 11:07:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 11:07:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/05/07 11:07:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:34936->10.242.41.137:8301: i/o timeout
2021/05/07 11:07:42 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:07:44 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-x5pmr)
2021/05/07 11:07:45 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:08:01 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:08:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:35210->10.242.41.137:8301: i/o timeout
2021/05/07 11:08:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:08:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:47028->10.242.41.146:8301: i/o timeout
2021/05/07 11:08:02 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:08:05 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:08:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:35248->10.242.41.137:8301: i/o timeout
2021/05/07 11:08:05 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:08:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:08:06 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500319205s
2021/05/07 11:08:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:35302->10.242.41.137:8301: i/o timeout
2021/05/07 11:08:07 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:08:09 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:08:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:35354->10.242.41.137:8301: i/o timeout
2021/05/07 11:08:11 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:08:12 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:08:12 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:08:12 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:08:13 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:08:13 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:08:13 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-x5pmr)
2021/05/07 11:08:19 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:08:19 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 11:08:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:35610->10.242.41.137:8301: i/o timeout
2021/05/07 11:08:28 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:08:29 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:08:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:08:31 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500202092s
2021/05/07 11:08:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:35632->10.242.41.137:8301: i/o timeout
2021/05/07 11:08:31 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:08:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:08:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:47464->10.242.41.146:8301: i/o timeout
2021/05/07 11:08:33 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:08:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:08:34 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.977350238s
2021/05/07 11:08:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:50350->10.242.41.146:8302: i/o timeout
2021/05/07 11:08:34 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:08:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:47520->10.242.41.146:8301: i/o timeout
2021/05/07 11:08:35 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:08:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:08:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:35718->10.242.41.137:8301: i/o timeout
2021/05/07 11:08:37 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:08:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:08:38 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500256949s
2021/05/07 11:08:39 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:08:39 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:08:39 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:08:41 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:08:41 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 11:08:45 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:08:46 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-x5pmr)
2021/05/07 11:08:46 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:08:46 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:08:53 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:08:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:08:54 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500202821s
2021/05/07 11:08:55 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/07 11:08:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:36038->10.242.41.137:8301: i/o timeout
2021/05/07 11:08:57 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:08:59 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:09:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:09:01 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 11:09:45 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 11:10:39 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:10:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:10:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:37662->10.242.41.137:8301: i/o timeout
2021/05/07 11:10:41 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:10:43 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:10:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:10:44 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-x5pmr but other probes failed, network may be misconfigured
2021/05/07 11:10:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:10:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:49592->10.242.41.146:8301: i/o timeout
2021/05/07 11:10:48 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:10:48 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:10:48 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:10:48 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:10:48 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:10:49 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:10:49 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:10:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:10:50 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500303127s
2021/05/07 11:10:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:37796->10.242.41.137:8301: i/o timeout
2021/05/07 11:10:51 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:10:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:10:53 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.967418739s
2021/05/07 11:10:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:10:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:52534->10.242.41.146:8302: i/o timeout
2021/05/07 11:10:54 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:10:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:37840->10.242.41.137:8301: i/o timeout
2021/05/07 11:10:54 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:10:55 [INFO] raft: aborting pipeline replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:10:55 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:10:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:10:56 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8301->10.242.41.146:55828: i/o timeout from=10.242.41.146:55828
2021/05/07 11:10:56 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8301->10.242.41.146:55828: i/o timeout from=10.242.41.146:55828
2021/05/07 11:10:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:37922->10.242.41.137:8301: i/o timeout
2021/05/07 11:10:57 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:10:58 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500552149s
2021/05/07 11:10:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:10:59 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:34876->10.242.41.146:8300: i/o timeout
2021/05/07 11:10:59 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:11:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:00 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.996670856s
2021/05/07 11:11:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:37956->10.242.41.137:8301: i/o timeout
2021/05/07 11:11:00 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:11:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:02 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:11:02 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:11:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:37974->10.242.41.137:8301: i/o timeout
2021/05/07 11:11:02 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:11:03 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.470328655s
2021/05/07 11:11:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:52674->10.242.41.146:8302: i/o timeout
2021/05/07 11:11:04 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:11:05 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:55076->10.242.41.146:8300: i/o timeout
2021/05/07 11:11:05 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:11:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:07 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:11:07 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:11:09 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:11:09 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:55122->10.242.41.146:8300: i/o timeout
2021/05/07 11:11:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:10 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:11:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:52812->10.242.41.146:8302: i/o timeout
2021/05/07 11:11:14 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:11:15 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:55216->10.242.41.146:8300: i/o timeout
2021/05/07 11:11:15 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:11:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:16 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:11:16 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:11:16 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:11:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:50034->10.242.41.146:8301: i/o timeout
2021/05/07 11:11:18 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:11:19 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:11:19 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.146:38262
2021/05/07 11:11:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:20 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:55266->10.242.41.146:8300: i/o timeout
2021/05/07 11:11:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:50060->10.242.41.146:8301: i/o timeout
2021/05/07 11:11:20 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:11:20 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:11:20 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:11:20 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:11:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:50078->10.242.41.146:8301: i/o timeout
2021/05/07 11:11:22 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:11:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:52904->10.242.41.146:8302: i/o timeout
2021/05/07 11:11:24 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:11:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:50122->10.242.41.146:8301: i/o timeout
2021/05/07 11:11:24 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:11:24 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:11:24 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:11:25 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8301->10.242.41.137:49090: i/o timeout from=10.242.41.137:49090
2021/05/07 11:11:25 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8301->10.242.41.137:49090: i/o timeout from=10.242.41.137:49090
2021/05/07 11:11:25 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:55364->10.242.41.146:8300: i/o timeout
2021/05/07 11:11:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:26 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8301->10.242.41.137:49092: i/o timeout from=10.242.41.137:49092
2021/05/07 11:11:26 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8301->10.242.41.137:49092: i/o timeout from=10.242.41.137:49092
2021/05/07 11:11:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:50184->10.242.41.146:8301: i/o timeout
2021/05/07 11:11:28 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:11:29 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:11:29 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:11:29 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:11:29 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:11:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:31 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:55440->10.242.41.146:8300: i/o timeout
2021/05/07 11:11:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:32 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:11:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:35 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:55522->10.242.41.146:8300: i/o timeout
2021/05/07 11:11:35 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:11:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:42 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:55594->10.242.41.146:8300: i/o timeout
2021/05/07 11:11:43 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:11:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:45 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:55664->10.242.41.146:8300: i/o timeout
2021/05/07 11:11:45 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:11:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:48 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:11:48 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:11:51 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:11:51 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 11:11:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:53 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:55776->10.242.41.146:8300: i/o timeout
2021/05/07 11:11:53 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:11:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:55 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:55858->10.242.41.146:8300: i/o timeout
2021/05/07 11:11:56 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:11:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:11:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:03 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:55958->10.242.41.146:8300: i/o timeout
2021/05/07 11:12:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:04 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:12:05 [DEBUG] raft-net: 10.242.139.204:8300 accepted connection from: 10.242.41.146:38010
2021/05/07 11:12:05 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 11:12:05 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:12:05 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:12:05 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:37586
2021/05/07 11:12:06 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:12:06 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:55994->10.242.41.146:8300: i/o timeout
2021/05/07 11:12:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:06 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:12:07 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500229244s
2021/05/07 11:12:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:10 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.948212059s
2021/05/07 11:12:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:12 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.384790708s
2021/05/07 11:12:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:16 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:56128->10.242.41.146:8300: i/o timeout
2021/05/07 11:12:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:16 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:56140->10.242.41.146:8300: i/o timeout
2021/05/07 11:12:16 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:12:17 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:12:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:22 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8302->10.242.41.146:54396: i/o timeout from=10.242.41.146:54396
2021/05/07 11:12:22 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8302->10.242.41.146:54396: i/o timeout from=10.242.41.146:54396
2021/05/07 11:12:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:24 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:12:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:26 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:56276->10.242.41.146:8300: i/o timeout
2021/05/07 11:12:27 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:56278->10.242.41.146:8300: i/o timeout
2021/05/07 11:12:27 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:12:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:28 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:12:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:30 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:12:30 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:12:33 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:12:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:37 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:56418->10.242.41.146:8300: i/o timeout
2021/05/07 11:12:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:38 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:12:38 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:56426->10.242.41.146:8300: i/o timeout
2021/05/07 11:12:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:40 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:12:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:44 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:12:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:48 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:56552->10.242.41.146:8300: i/o timeout
2021/05/07 11:12:49 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:12:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:51 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:56580->10.242.41.146:8300: i/o timeout
2021/05/07 11:12:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:56 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:12:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:12:59 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:56696->10.242.41.146:8300: i/o timeout
2021/05/07 11:13:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:00 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:13:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:06 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:56792->10.242.41.146:8300: i/o timeout
2021/05/07 11:13:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:10 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:56840->10.242.41.146:8300: i/o timeout
2021/05/07 11:13:11 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:13:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:12 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:13:12 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:13:15 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:13:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:16 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:13:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:21 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:56974->10.242.41.146:8300: i/o timeout
2021/05/07 11:13:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:22 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:13:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:24 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:13:24 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:13:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:26 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:57058->10.242.41.146:8300: i/o timeout
2021/05/07 11:13:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:32 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:57126->10.242.41.146:8300: i/o timeout
2021/05/07 11:13:33 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:13:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:36 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:13:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:43 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:57278->10.242.41.146:8300: i/o timeout
2021/05/07 11:13:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:46 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:13:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:46 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:57322->10.242.41.146:8300: i/o timeout
2021/05/07 11:13:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:54 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:13:54 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:13:56 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:57446->10.242.41.146:8300: i/o timeout
2021/05/07 11:13:56 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:13:57 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:13:57 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/07 11:13:57 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 11:13:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:13:59 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:14:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:04 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:14:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:06 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:57586->10.242.41.146:8300: i/o timeout
2021/05/07 11:14:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:09 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:57618->10.242.41.146:8300: i/o timeout
2021/05/07 11:14:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:15 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:14:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:17 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:14:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:25 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:57842->10.242.41.146:8300: i/o timeout
2021/05/07 11:14:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:27 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:57860->10.242.41.146:8300: i/o timeout
2021/05/07 11:14:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:36 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:14:36 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:14:36 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:14:37 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:14:39 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:14:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:44 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:14:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:46 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:58114->10.242.41.146:8300: i/o timeout
2021/05/07 11:14:47 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:58122->10.242.41.146:8300: i/o timeout
2021/05/07 11:14:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:50 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:14:50 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:14:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:41158->10.242.41.137:8301: i/o timeout
2021/05/07 11:14:51 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:14:51 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:14:51 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:14:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:41174->10.242.41.137:8301: i/o timeout
2021/05/07 11:14:53 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:14:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:55 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:14:55 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:14:55 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:14:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:41220->10.242.41.137:8301: i/o timeout
2021/05/07 11:14:55 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:14:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:14:57 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:14:57 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:14:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:55992->10.242.41.146:8302: i/o timeout
2021/05/07 11:15:04 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:15:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:07 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:58410->10.242.41.146:8300: i/o timeout
2021/05/07 11:15:07 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:58412->10.242.41.146:8300: i/o timeout
2021/05/07 11:15:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:56064->10.242.41.146:8302: i/o timeout
2021/05/07 11:15:14 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:15:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:17 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:15:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:18 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:15:18 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:15:18 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:15:21 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:15:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:24 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:15:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:27 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:58680->10.242.41.146:8300: i/o timeout
2021/05/07 11:15:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:28 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:58690->10.242.41.146:8300: i/o timeout
2021/05/07 11:15:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:56332->10.242.41.146:8302: i/o timeout
2021/05/07 11:15:29 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:15:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:34 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:15:34 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:15:34 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:15:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:38 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:15:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:39 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:15:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:56454->10.242.41.146:8302: i/o timeout
2021/05/07 11:15:39 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:15:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:41 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:15:41 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:15:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:41894->10.242.41.137:8301: i/o timeout
2021/05/07 11:15:42 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:15:43 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:15:43 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:15:43 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:15:43 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:15:43 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:15:43 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:15:43 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-x5pmr)
2021/05/07 11:15:43 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:15:44 [INFO] raft: pipelining replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:15:44 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:15:44 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:15:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:41966->10.242.41.137:8301: i/o timeout
2021/05/07 11:15:46 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:15:47 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:15:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:42018->10.242.41.137:8301: i/o timeout
2021/05/07 11:15:50 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:15:51 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:15:55 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:15:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:42136->10.242.41.137:8301: i/o timeout
2021/05/07 11:15:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:15:59 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-x5pmr but other probes failed, network may be misconfigured
2021/05/07 11:16:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:16:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:16:02 [ERR] raft: Failed to pipeline AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: write tcp 10.242.139.204:58942->10.242.41.146:8300: use of closed network connection
2021/05/07 11:16:02 [INFO] raft: aborting pipeline replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:16:03 [INFO] raft: pipelining replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:16:03 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:16:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:16:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:42228->10.242.41.137:8301: i/o timeout
2021/05/07 11:16:04 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:16:07 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:16:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:42276->10.242.41.137:8301: i/o timeout
2021/05/07 11:16:07 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:16:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:16:09 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:16:09 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:16:10 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:16:10 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:16:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:42366->10.242.41.137:8301: i/o timeout
2021/05/07 11:16:14 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:16:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:54266->10.242.41.146:8301: i/o timeout
2021/05/07 11:16:20 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:16:20 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500704849s
2021/05/07 11:16:21 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:16:33 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:16:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:54450->10.242.41.146:8301: i/o timeout
2021/05/07 11:16:33 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:16:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:16:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:16:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:54490->10.242.41.146:8301: i/o timeout
2021/05/07 11:16:36 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:16:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:42698->10.242.41.137:8301: i/o timeout
2021/05/07 11:16:39 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:16:41 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:16:41 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:16:41 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:16:41 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:16:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:16:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:42744->10.242.41.137:8301: i/o timeout
2021/05/07 11:16:42 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:16:43 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500308501s
2021/05/07 11:16:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:16:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:16:46 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.931777572s
2021/05/07 11:16:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:54606->10.242.41.146:8301: i/o timeout
2021/05/07 11:16:46 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:16:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:16:48 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.406384119s
2021/05/07 11:16:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:57538->10.242.41.146:8302: i/o timeout
2021/05/07 11:16:49 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:16:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:54664->10.242.41.146:8301: i/o timeout
2021/05/07 11:16:49 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:16:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:16:51 [INFO] raft: aborting pipeline replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:16:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:54706->10.242.41.146:8301: i/o timeout
2021/05/07 11:16:51 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:16:52 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:59376->10.242.41.146:8300: i/o timeout
2021/05/07 11:16:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:16:53 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500252903s
2021/05/07 11:16:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:54728->10.242.41.146:8301: i/o timeout
2021/05/07 11:16:53 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:16:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:16:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:16:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:54774->10.242.41.146:8301: i/o timeout
2021/05/07 11:16:56 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:16:57 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 11:16:58 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:16:58 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:16:58 [INFO] raft: pipelining replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:16:58 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-x5pmr)
2021/05/07 11:17:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:43134->10.242.41.137:8301: i/o timeout
2021/05/07 11:17:09 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:17:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:43186->10.242.41.137:8301: i/o timeout
2021/05/07 11:17:13 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:17:13 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:17:13 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:17:13 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:17:14 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:17:14 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:17:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:43442->10.242.41.137:8301: i/o timeout
2021/05/07 11:17:31 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:17:33 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:17:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:17:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:43460->10.242.41.137:8301: i/o timeout
2021/05/07 11:17:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:17:35 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:17:35 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:17:35 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:17:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:17:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:43510->10.242.41.137:8301: i/o timeout
2021/05/07 11:17:36 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:17:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:17:38 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500224566s
2021/05/07 11:17:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:55358->10.242.41.146:8301: i/o timeout
2021/05/07 11:17:39 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:17:41 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:17:41 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.992591325s
2021/05/07 11:17:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:17:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:55398->10.242.41.146:8301: i/o timeout
2021/05/07 11:17:42 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:17:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:17:44 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.5002165s
2021/05/07 11:17:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:58280->10.242.41.146:8302: i/o timeout
2021/05/07 11:17:44 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:17:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:55412->10.242.41.146:8301: i/o timeout
2021/05/07 11:17:44 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:17:47 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:17:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:17:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:55490->10.242.41.146:8301: i/o timeout
2021/05/07 11:17:48 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:17:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:17:50 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500205422s
2021/05/07 11:17:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:55512->10.242.41.146:8301: i/o timeout
2021/05/07 11:17:50 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:17:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:17:52 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.905863282s
2021/05/07 11:17:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:55552->10.242.41.146:8301: i/o timeout
2021/05/07 11:17:53 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:17:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:17:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:58418->10.242.41.146:8302: i/o timeout
2021/05/07 11:17:54 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:17:55 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.341058896s
2021/05/07 11:17:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:55598->10.242.41.146:8301: i/o timeout
2021/05/07 11:17:55 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:17:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:17:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:55620->10.242.41.146:8301: i/o timeout
2021/05/07 11:17:57 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:17:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:17:58 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:60082->10.242.41.146:8300: i/o timeout
2021/05/07 11:17:59 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500240245s
2021/05/07 11:18:00 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:18:00 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:18:00 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:18:00 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:18:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:55662->10.242.41.146:8301: i/o timeout
2021/05/07 11:18:00 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:18:01 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:18:01 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 11:18:02 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.972982702s
2021/05/07 11:18:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:04 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.420470881s
2021/05/07 11:18:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:58568->10.242.41.146:8302: i/o timeout
2021/05/07 11:18:04 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:18:04 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:18:05 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8301->10.242.41.146:60116: i/o timeout from=10.242.41.146:60116
2021/05/07 11:18:05 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8301->10.242.41.146:60116: i/o timeout from=10.242.41.146:60116
2021/05/07 11:18:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:07 [INFO] raft: aborting pipeline replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:18:07 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:18:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:09 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:32778->10.242.41.146:8300: i/o timeout
2021/05/07 11:18:09 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.507583235s
2021/05/07 11:18:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:10 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:18:12 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.948079755s
2021/05/07 11:18:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:14 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.352975183s
2021/05/07 11:18:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:58704->10.242.41.146:8302: i/o timeout
2021/05/07 11:18:14 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:18:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:17 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:32884->10.242.41.146:8300: i/o timeout
2021/05/07 11:18:17 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:18:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:20 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:18:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:22 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500161407s
2021/05/07 11:18:23 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:18:23 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:18:23 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:18:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:24 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:18:24 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:18:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:58782->10.242.41.146:8302: i/o timeout
2021/05/07 11:18:24 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:18:24 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:18:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:55984->10.242.41.146:8301: i/o timeout
2021/05/07 11:18:24 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:18:25 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500328333s
2021/05/07 11:18:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:56018->10.242.41.146:8301: i/o timeout
2021/05/07 11:18:26 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:18:27 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:33018->10.242.41.146:8300: i/o timeout
2021/05/07 11:18:27 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.954187556s
2021/05/07 11:18:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:56032->10.242.41.146:8301: i/o timeout
2021/05/07 11:18:28 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:18:28 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:18:28 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:18:28 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:18:28 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:18:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:30 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.39566749s
2021/05/07 11:18:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:33 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:32922->10.242.41.146:8300: i/o timeout
2021/05/07 11:18:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:34 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:18:35 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:18:35 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:18:37 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:18:37 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:33158->10.242.41.146:8300: i/o timeout
2021/05/07 11:18:37 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:18:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:44 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:33244->10.242.41.146:8300: i/o timeout
2021/05/07 11:18:44 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:18:45 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:18:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:47 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:33288->10.242.41.146:8300: i/o timeout
2021/05/07 11:18:47 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:18:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:54 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:18:55 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:33414->10.242.41.146:8300: i/o timeout
2021/05/07 11:18:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:18:56 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:18:57 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:33430->10.242.41.146:8300: i/o timeout
2021/05/07 11:18:57 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:18:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:06 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:33548->10.242.41.146:8300: i/o timeout
2021/05/07 11:19:07 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:19:07 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:33560->10.242.41.146:8300: i/o timeout
2021/05/07 11:19:07 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:19:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:16 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:19:16 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:19:17 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:33694->10.242.41.146:8300: i/o timeout
2021/05/07 11:19:17 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:33698->10.242.41.146:8300: i/o timeout
2021/05/07 11:19:17 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:19:17 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:19:19 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:19:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:23 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:19:23 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:19:23 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:19:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:24 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:19:24 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:19:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:56784->10.242.41.146:8301: i/o timeout
2021/05/07 11:19:24 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:19:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:56828->10.242.41.146:8301: i/o timeout
2021/05/07 11:19:26 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:19:27 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:33834->10.242.41.146:8300: i/o timeout
2021/05/07 11:19:27 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:33836->10.242.41.146:8300: i/o timeout
2021/05/07 11:19:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:28 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:19:28 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:19:28 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:19:28 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:19:29 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:19:29 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:19:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:59722->10.242.41.146:8302: i/o timeout
2021/05/07 11:19:29 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:19:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:56864->10.242.41.146:8301: i/o timeout
2021/05/07 11:19:29 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:19:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:45068->10.242.41.137:8301: i/o timeout
2021/05/07 11:19:32 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:19:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:34 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:19:34 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:19:34 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:19:35 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-x5pmr)
2021/05/07 11:19:35 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-x5pmr but other probes failed, network may be misconfigured
2021/05/07 11:19:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:38 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:33982->10.242.41.146:8300: i/o timeout
2021/05/07 11:19:38 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:33984->10.242.41.146:8300: i/o timeout
2021/05/07 11:19:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:56968->10.242.41.146:8301: i/o timeout
2021/05/07 11:19:38 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:19:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:59784->10.242.41.146:8302: i/o timeout
2021/05/07 11:19:39 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:19:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:45222->10.242.41.137:8301: i/o timeout
2021/05/07 11:19:42 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:19:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:44 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:19:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:45302->10.242.41.137:8301: i/o timeout
2021/05/07 11:19:45 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:19:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:46 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 11:19:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:57122->10.242.41.146:8301: i/o timeout
2021/05/07 11:19:47 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:19:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:48 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:19:48 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:19:48 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:19:49 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:34140->10.242.41.146:8300: i/o timeout
2021/05/07 11:19:49 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:34142->10.242.41.146:8300: i/o timeout
2021/05/07 11:19:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:50 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:19:50 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:19:50 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:19:50 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:19:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:57162->10.242.41.146:8301: i/o timeout
2021/05/07 11:19:50 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:19:51 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:19:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:52 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:19:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:54 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:19:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:19:59 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 11:19:59 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:19:59 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:19:59 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:20:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:01 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:34298->10.242.41.146:8300: i/o timeout
2021/05/07 11:20:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:02 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:34318->10.242.41.146:8300: i/o timeout
2021/05/07 11:20:03 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:20:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:60210->10.242.41.146:8302: i/o timeout
2021/05/07 11:20:04 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:20:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:07 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:20:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:13 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:34468->10.242.41.146:8300: i/o timeout
2021/05/07 11:20:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:16 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:20:17 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:34534->10.242.41.146:8300: i/o timeout
2021/05/07 11:20:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:19 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:20:19 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:20:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:45762->10.242.41.137:8301: i/o timeout
2021/05/07 11:20:20 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:20:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:45794->10.242.41.137:8301: i/o timeout
2021/05/07 11:20:23 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:20:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:45846->10.242.41.137:8301: i/o timeout
2021/05/07 11:20:25 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:20:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:26 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:34650->10.242.41.146:8300: i/o timeout
2021/05/07 11:20:27 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:20:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:45854->10.242.41.137:8301: i/o timeout
2021/05/07 11:20:27 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:20:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:45900->10.242.41.137:8301: i/o timeout
2021/05/07 11:20:30 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:20:31 [INFO] raft: pipelining replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:20:31 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:20:32 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:20:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:45918->10.242.41.137:8301: i/o timeout
2021/05/07 11:20:32 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:20:33 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:20:34 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500245481s
2021/05/07 11:20:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:36 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.95159456s
2021/05/07 11:20:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:45986->10.242.41.137:8301: i/o timeout
2021/05/07 11:20:36 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:20:37 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:20:37 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:20:37 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:20:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:45998->10.242.41.137:8301: i/o timeout
2021/05/07 11:20:38 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:20:38 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.401921602s
2021/05/07 11:20:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:40 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8301->10.242.41.146:33238: i/o timeout from=10.242.41.146:33238
2021/05/07 11:20:40 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8301->10.242.41.146:33238: i/o timeout from=10.242.41.146:33238
2021/05/07 11:20:40 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:20:40 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:20:41 [ERR] raft: Failed to pipeline AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: write tcp 10.242.139.204:34814->10.242.41.146:8300: use of closed network connection
2021/05/07 11:20:41 [INFO] raft: aborting pipeline replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:20:41 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:20:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:42 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:34862->10.242.41.146:8300: i/o timeout
2021/05/07 11:20:44 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500296808s
2021/05/07 11:20:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:46 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.957725848s
2021/05/07 11:20:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:49 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.411709909s
2021/05/07 11:20:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:50 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:20:50 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:45624
2021/05/07 11:20:51 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:34994->10.242.41.146:8300: i/o timeout
2021/05/07 11:20:51 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:20:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:53 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:20:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:60892->10.242.41.146:8302: i/o timeout
2021/05/07 11:20:54 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:20:55 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:20:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:20:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:21:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:21:01 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:35132->10.242.41.146:8300: i/o timeout
2021/05/07 11:21:01 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:21:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:21:03 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:35164->10.242.41.146:8300: i/o timeout
2021/05/07 11:21:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:21:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:21:07 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 11:21:07 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:21:07 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:21:07 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:21:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:21:08 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:21:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:21:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:58254->10.242.41.146:8301: i/o timeout
2021/05/07 11:21:10 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:21:11 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:35270->10.242.41.146:8300: i/o timeout
2021/05/07 11:21:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:21:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:58298->10.242.41.146:8301: i/o timeout
2021/05/07 11:21:13 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:21:15 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:21:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:58314->10.242.41.146:8301: i/o timeout
2021/05/07 11:21:15 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:21:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:21:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:58362->10.242.41.146:8301: i/o timeout
2021/05/07 11:21:17 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:21:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:21:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:58390->10.242.41.146:8301: i/o timeout
2021/05/07 11:21:19 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:21:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:21:21 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:35404->10.242.41.146:8300: i/o timeout
2021/05/07 11:21:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:21:22 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:21:23 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:21:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:58440->10.242.41.146:8301: i/o timeout
2021/05/07 11:21:23 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:21:23 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:21:23 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:21:23 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:21:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:21:24 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:35452->10.242.41.146:8300: i/o timeout
2021/05/07 11:21:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:21:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:21:29 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:21:29 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:21:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:33160->10.242.41.146:8302: i/o timeout
2021/05/07 11:21:29 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:21:31 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:35548->10.242.41.146:8300: i/o timeout
2021/05/07 11:21:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:46742->10.242.41.137:8301: i/o timeout
2021/05/07 11:21:31 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:21:31 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:21:33 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:21:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:46752->10.242.41.137:8301: i/o timeout
2021/05/07 11:21:33 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:21:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:21:35 [INFO] raft: pipelining replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:21:35 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:21:35 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:21:35 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:21:36 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:21:36 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:21:36 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:21:36 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:21:39 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:21:39 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 11:21:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:46864->10.242.41.137:8301: i/o timeout
2021/05/07 11:21:39 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:21:43 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:21:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:58720->10.242.41.146:8301: i/o timeout
2021/05/07 11:21:43 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:21:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:21:45 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500237049s
2021/05/07 11:21:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:21:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:58740->10.242.41.146:8301: i/o timeout
2021/05/07 11:21:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:46980->10.242.41.137:8301: i/o timeout
2021/05/07 11:21:48 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:21:49 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:21:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:21:50 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:21:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:47032->10.242.41.137:8301: i/o timeout
2021/05/07 11:21:51 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:21:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:21:52 [INFO] raft: aborting pipeline replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:21:53 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:21:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:21:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:47078->10.242.41.137:8301: i/o timeout
2021/05/07 11:21:54 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:21:55 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500292835s
2021/05/07 11:21:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:21:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:58932->10.242.41.146:8301: i/o timeout
2021/05/07 11:21:57 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:21:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:21:59 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500254201s
2021/05/07 11:21:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:47136->10.242.41.137:8301: i/o timeout
2021/05/07 11:21:59 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:22:00 [INFO] raft: pipelining replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:22:01 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:22:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:22:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:47184->10.242.41.137:8301: i/o timeout
2021/05/07 11:22:02 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:22:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:22:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:59008->10.242.41.146:8301: i/o timeout
2021/05/07 11:22:05 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:22:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:22:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:22:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:47252->10.242.41.137:8301: i/o timeout
2021/05/07 11:22:08 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:22:09 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500302006s
2021/05/07 11:22:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:22:11 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:22:11 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:22:11 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.961289342s
2021/05/07 11:22:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:47308->10.242.41.137:8301: i/o timeout
2021/05/07 11:22:11 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:22:13 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:22:13 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:22:13 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:22:13 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:59132->10.242.41.146:8301: i/o timeout
2021/05/07 11:22:13 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:22:14 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.393385272s
2021/05/07 11:22:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:22:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:33784->10.242.41.146:8302: i/o timeout
2021/05/07 11:22:14 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:22:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:59170->10.242.41.146:8301: i/o timeout
2021/05/07 11:22:15 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:22:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:22:16 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:22:16 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:22:16 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:22:16 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:22:16 [ERR] raft: Failed to pipeline AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: write tcp 10.242.139.204:35992->10.242.41.146:8300: use of closed network connection
2021/05/07 11:22:16 [INFO] raft: aborting pipeline replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:22:16 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:22:17 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:22:18 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:22:18 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:22:18 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:22:18 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 11:22:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:22:18 [ERR] raft: peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300} has newer term, stopping replication
2021/05/07 11:22:18 [INFO] raft: Node at 10.242.139.204:8300 [Follower] entering Follower state (Leader: "")
2021/05/07 11:22:18 [INFO] raft: aborting pipeline replication to peer {Voter dff60f1e-22b9-7f6a-153b-e2e4aea758bb 10.242.113.198:8300}
2021/05/07 11:22:18 [INFO] consul: cluster leadership lost
2021/05/07 11:22:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:59244->10.242.41.146:8301: i/o timeout
2021/05/07 11:22:19 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:22:21 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:22:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:47476->10.242.41.137:8301: i/o timeout
2021/05/07 11:22:22 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:22:24 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since our last index is greater (741, 736)
2021/05/07 11:22:26 [DEBUG] raft-net: 10.242.139.204:8300 accepted connection from: 10.242.113.198:47630
2021/05/07 11:22:26 [INFO] consul: New leader elected: dev-consul-server-0
2021/05/07 11:22:26 [DEBUG] raft-net: 10.242.139.204:8300 accepted connection from: 10.242.113.198:47656
2021/05/07 11:22:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:47698->10.242.41.137:8301: i/o timeout
2021/05/07 11:22:39 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:23:40 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-x5pmr)
2021/05/07 11:23:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:48554->10.242.41.137:8301: i/o timeout
2021/05/07 11:23:42 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:23:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:48640->10.242.41.137:8301: i/o timeout
2021/05/07 11:23:47 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:23:50 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:23:51 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:23:51 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/07 11:23:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:60508->10.242.41.146:8301: i/o timeout
2021/05/07 11:23:52 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:23:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:60536->10.242.41.146:8301: i/o timeout
2021/05/07 11:23:55 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:23:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:60576->10.242.41.146:8301: i/o timeout
2021/05/07 11:23:57 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:24:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:60624->10.242.41.146:8301: i/o timeout
2021/05/07 11:24:00 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:24:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:60646->10.242.41.146:8301: i/o timeout
2021/05/07 11:24:02 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:24:04 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:24:04 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:24:04 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:24:04 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 11:24:04 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:24:06 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:24:12 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8301->10.242.41.137:59588: i/o timeout from=10.242.41.137:59588
2021/05/07 11:24:12 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8301->10.242.41.137:59588: i/o timeout from=10.242.41.137:59588
2021/05/07 11:24:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:35424->10.242.41.146:8302: i/o timeout
2021/05/07 11:24:14 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:24:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:35552->10.242.41.146:8302: i/o timeout
2021/05/07 11:24:24 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:24:38 [ERR] yamux: keepalive failed: session shutdown
2021/05/07 11:24:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:35744->10.242.41.146:8302: i/o timeout
2021/05/07 11:24:39 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:24:40 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:24:40 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:47840
2021/05/07 11:24:44 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:24:44 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:24:44 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:24:46 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:24:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:35884->10.242.41.146:8302: i/o timeout
2021/05/07 11:24:49 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:25:26 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:25:41 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8302->10.242.41.146:37182: i/o timeout from=10.242.41.146:37182
2021/05/07 11:25:41 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8302->10.242.41.146:37182: i/o timeout from=10.242.41.146:37182
2021/05/07 11:25:42 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:25:54 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:25:58 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:25:58 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:25:58 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:26:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:50348->10.242.41.137:8301: i/o timeout
2021/05/07 11:26:00 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:26:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:33944->10.242.41.146:8301: i/o timeout
2021/05/07 11:26:03 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:26:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:50396->10.242.41.137:8301: i/o timeout
2021/05/07 11:26:06 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:26:08 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.113.198:8300
2021/05/07 11:26:09 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:26:09 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:26:09 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:26:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:34054->10.242.41.146:8301: i/o timeout
2021/05/07 11:26:09 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:26:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:50520->10.242.41.137:8301: i/o timeout
2021/05/07 11:26:11 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:26:12 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:26:12 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:26:22 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:26:22 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:52402
2021/05/07 11:26:34 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 11:26:34 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:26:34 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:26:34 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:26:34 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:26:36 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:26:36 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:26:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:34452->10.242.41.146:8301: i/o timeout
2021/05/07 11:26:37 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:26:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:34486->10.242.41.146:8301: i/o timeout
2021/05/07 11:26:39 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:26:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:34526->10.242.41.146:8301: i/o timeout
2021/05/07 11:26:42 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:26:42 [INFO] consul: New leader elected: dev-consul-server-0
2021/05/07 11:26:43 [DEBUG] raft-net: 10.242.139.204:8300 accepted connection from: 10.242.113.198:50786
2021/05/07 11:26:44 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:26:44 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:26:45 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:26:49 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.113.198:8300
2021/05/07 11:26:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:34674->10.242.41.146:8301: i/o timeout
2021/05/07 11:26:53 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:26:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:34722->10.242.41.146:8301: i/o timeout
2021/05/07 11:26:56 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:26:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:34750->10.242.41.146:8301: i/o timeout
2021/05/07 11:26:58 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:26:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:37622->10.242.41.146:8302: i/o timeout
2021/05/07 11:26:59 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:27:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:34768->10.242.41.146:8301: i/o timeout
2021/05/07 11:27:00 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:27:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:34820->10.242.41.146:8301: i/o timeout
2021/05/07 11:27:03 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:27:05 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:27:05 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:27:05 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:27:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:34864->10.242.41.146:8301: i/o timeout
2021/05/07 11:27:05 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:27:06 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:27:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:37690->10.242.41.146:8302: i/o timeout
2021/05/07 11:27:09 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:27:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:37970->10.242.41.146:8302: i/o timeout
2021/05/07 11:27:24 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:27:28 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:27:28 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:27:29 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:27:29 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:27:29 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:27:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:35192->10.242.41.146:8301: i/o timeout
2021/05/07 11:27:30 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:27:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:35218->10.242.41.146:8301: i/o timeout
2021/05/07 11:27:32 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:27:33 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:27:33 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:27:33 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:27:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:38104->10.242.41.146:8302: i/o timeout
2021/05/07 11:27:34 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:27:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:35244->10.242.41.146:8301: i/o timeout
2021/05/07 11:27:34 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:27:46 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:27:50 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:27:50 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:27:50 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:27:50 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:27:50 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:27:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:35502->10.242.41.146:8301: i/o timeout
2021/05/07 11:27:53 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:28:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:52044->10.242.41.137:8301: i/o timeout
2021/05/07 11:28:01 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:28:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:52144->10.242.41.137:8301: i/o timeout
2021/05/07 11:28:09 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:28:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:52306->10.242.41.137:8301: i/o timeout
2021/05/07 11:28:20 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:28:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:35902->10.242.41.146:8301: i/o timeout
2021/05/07 11:28:23 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:28:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:52398->10.242.41.137:8301: i/o timeout
2021/05/07 11:28:26 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:28:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:52418->10.242.41.137:8301: i/o timeout
2021/05/07 11:28:28 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:28:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:36018->10.242.41.146:8301: i/o timeout
2021/05/07 11:28:31 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:28:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:36052->10.242.41.146:8301: i/o timeout
2021/05/07 11:28:33 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:28:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:52494->10.242.41.137:8301: i/o timeout
2021/05/07 11:28:35 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:28:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:36108->10.242.41.146:8301: i/o timeout
2021/05/07 11:28:37 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:28:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:52566->10.242.41.137:8301: i/o timeout
2021/05/07 11:28:39 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:28:41 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:28:41 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:28:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:36156->10.242.41.146:8301: i/o timeout
2021/05/07 11:28:41 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:28:43 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:28:43 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:28:43 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:28:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:39144->10.242.41.146:8302: i/o timeout
2021/05/07 11:28:49 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:28:50 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:28:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:39200->10.242.41.146:8302: i/o timeout
2021/05/07 11:28:59 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:29:09 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 11:29:10 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:29:10 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:29:10 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:29:17 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:29:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:53092->10.242.41.137:8301: i/o timeout
2021/05/07 11:29:18 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:29:19 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:29:19 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:29:19 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:29:20 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:29:20 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.113.198:8300
2021/05/07 11:29:20 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:29:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:36734->10.242.41.146:8301: i/o timeout
2021/05/07 11:29:23 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:29:25 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/07 11:29:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:36752->10.242.41.146:8301: i/o timeout
2021/05/07 11:29:25 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:29:26 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:29:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:53220->10.242.41.137:8301: i/o timeout
2021/05/07 11:29:28 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:29:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:53252->10.242.41.137:8301: i/o timeout
2021/05/07 11:29:31 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:29:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:39744->10.242.41.146:8302: i/o timeout
2021/05/07 11:29:34 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:29:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:36860->10.242.41.146:8301: i/o timeout
2021/05/07 11:29:34 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:29:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:36912->10.242.41.146:8301: i/o timeout
2021/05/07 11:29:37 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:29:39 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:29:39 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:29:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:53364->10.242.41.137:8301: i/o timeout
2021/05/07 11:29:40 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:29:42 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:29:42 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:29:42 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:29:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:39880->10.242.41.146:8302: i/o timeout
2021/05/07 11:29:44 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:29:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:40012->10.242.41.146:8302: i/o timeout
2021/05/07 11:29:54 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:30:00 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:30:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:40270->10.242.41.146:8302: i/o timeout
2021/05/07 11:30:14 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:30:24 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:30:24 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:55796
2021/05/07 11:30:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:40476->10.242.41.146:8302: i/o timeout
2021/05/07 11:30:29 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:30:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:40608->10.242.41.146:8302: i/o timeout
2021/05/07 11:30:39 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:30:40 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:30:40 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:30:40 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:30:40 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:30:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:40706->10.242.41.146:8302: i/o timeout
2021/05/07 11:30:49 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:30:51 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-6xm62 10.242.113.229
2021/05/07 11:31:01 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8302->10.242.41.146:41198: i/o timeout from=10.242.41.146:41198
2021/05/07 11:31:01 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8302->10.242.41.146:41198: i/o timeout from=10.242.41.146:41198
2021/05/07 11:31:09 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:31:09 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:31:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:38442->10.242.41.146:8301: i/o timeout
2021/05/07 11:31:10 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:31:12 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:31:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:38498->10.242.41.146:8301: i/o timeout
2021/05/07 11:31:14 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:31:15 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:31:15 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:31:24 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:31:26 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.113.198:8300
2021/05/07 11:31:32 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8301->10.242.41.146:43118: i/o timeout from=10.242.41.146:43118
2021/05/07 11:31:32 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8301->10.242.41.146:43118: i/o timeout from=10.242.41.146:43118
2021/05/07 11:31:50 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:32:07 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:32:07 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:32:07 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 11:32:07 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:32:07 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:32:07 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:32:14 [INFO] consul: New leader elected: dev-consul-server-0
2021/05/07 11:32:15 [DEBUG] raft-net: 10.242.139.204:8300 accepted connection from: 10.242.113.198:55452
2021/05/07 11:32:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:39508->10.242.41.146:8301: i/o timeout
2021/05/07 11:32:15 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:32:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:55998->10.242.41.137:8301: i/o timeout
2021/05/07 11:32:20 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:32:24 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:32:24 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:32:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:42504->10.242.41.146:8302: i/o timeout
2021/05/07 11:32:24 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:32:25 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.113.198:8300
2021/05/07 11:32:26 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:32:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:39714->10.242.41.146:8301: i/o timeout
2021/05/07 11:32:28 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:32:28 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:32:28 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:32:28 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:32:30 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:32:31 [WARN] raft: Heartbeat timeout from "10.242.113.198:8300" reached, starting election
2021/05/07 11:32:31 [INFO] raft: Node at 10.242.139.204:8300 [Candidate] entering Candidate state in term 6
2021/05/07 11:32:31 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:32:32 [INFO] raft: Election won. Tally: 2
2021/05/07 11:32:32 [INFO] raft: Node at 10.242.139.204:8300 [Leader] entering Leader state
2021/05/07 11:32:32 [INFO] consul: cluster leadership acquired
2021/05/07 11:32:32 [INFO] raft: Added peer dff60f1e-22b9-7f6a-153b-e2e4aea758bb, starting replication
2021/05/07 11:32:32 [INFO] raft: Added peer e90a9b84-7b02-fc1f-15b4-2419fe66fc38, starting replication
2021/05/07 11:32:32 [INFO] consul: New leader elected: dev-consul-server-2
2021/05/07 11:32:32 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:32:32 [INFO] raft: pipelining replication to peer {Voter dff60f1e-22b9-7f6a-153b-e2e4aea758bb 10.242.113.198:8300}
2021/05/07 11:32:32 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:32:33 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:32:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:42652->10.242.41.146:8302: i/o timeout
2021/05/07 11:32:34 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:32:35 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.501693404s
2021/05/07 11:32:35 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:32:35 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 11:32:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:32:37 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.971454012s
2021/05/07 11:32:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:32:39 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.158230517s
2021/05/07 11:32:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:32:41 [ERR] raft: Failed to make RequestVote RPC to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:35748->10.242.41.146:8300: i/o timeout
2021/05/07 11:32:42 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:36338->10.242.41.146:8300: i/o timeout
2021/05/07 11:32:42 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:32:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:32:43 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:45156->10.242.41.146:8300: i/o timeout
2021/05/07 11:32:44 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:32:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:32:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:42826->10.242.41.146:8302: i/o timeout
2021/05/07 11:32:44 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:32:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:32:48 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8301->10.242.41.137:37060: i/o timeout from=10.242.41.137:37060
2021/05/07 11:32:48 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8301->10.242.41.137:37060: i/o timeout from=10.242.41.137:37060
2021/05/07 11:32:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:32:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:32:52 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:45308->10.242.41.146:8300: i/o timeout
2021/05/07 11:32:52 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:32:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:32:54 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:45334->10.242.41.146:8300: i/o timeout
2021/05/07 11:32:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:32:54 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:32:54 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:32:54 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:32:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:42988->10.242.41.146:8302: i/o timeout
2021/05/07 11:32:54 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:32:54 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:32:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:32:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:02 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:45468->10.242.41.146:8300: i/o timeout
2021/05/07 11:33:02 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:33:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:04 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:45524->10.242.41.146:8300: i/o timeout
2021/05/07 11:33:05 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:33:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:10 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:33:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:12 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:45634->10.242.41.146:8300: i/o timeout
2021/05/07 11:33:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:12 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:33:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:14 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:33:14 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:33:15 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:45692->10.242.41.146:8300: i/o timeout
2021/05/07 11:33:16 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:33:17 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:33:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:22 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:45790->10.242.41.146:8300: i/o timeout
2021/05/07 11:33:22 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:33:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:26 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:45868->10.242.41.146:8300: i/o timeout
2021/05/07 11:33:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:27 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:33:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:32 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:45970->10.242.41.146:8300: i/o timeout
2021/05/07 11:33:33 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:33:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:36 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:33:36 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:33:37 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:46050->10.242.41.146:8300: i/o timeout
2021/05/07 11:33:38 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:33:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:57272->10.242.41.137:8301: i/o timeout
2021/05/07 11:33:38 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:33:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:57336->10.242.41.137:8301: i/o timeout
2021/05/07 11:33:41 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:33:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:42 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:33:42 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:33:42 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:33:43 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:46128->10.242.41.146:8300: i/o timeout
2021/05/07 11:33:43 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:33:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:47 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:33:47 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:33:48 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:46218->10.242.41.146:8300: i/o timeout
2021/05/07 11:33:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:49 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:33:50 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:33:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:53 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:46312->10.242.41.146:8300: i/o timeout
2021/05/07 11:33:54 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:33:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:43990->10.242.41.146:8302: i/o timeout
2021/05/07 11:33:54 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:33:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:33:56 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:33:56 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:33:57 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8301->10.242.41.146:44654: i/o timeout from=10.242.41.146:44654
2021/05/07 11:33:57 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8301->10.242.41.146:44654: i/o timeout from=10.242.41.146:44654
2021/05/07 11:33:59 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:46412->10.242.41.146:8300: i/o timeout
2021/05/07 11:33:59 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:34:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:00 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:34:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:04 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:46506->10.242.41.146:8300: i/o timeout
2021/05/07 11:34:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:44082->10.242.41.146:8302: i/o timeout
2021/05/07 11:34:04 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:34:05 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:34:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:10 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:46616->10.242.41.146:8300: i/o timeout
2021/05/07 11:34:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:13 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:34:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:44326->10.242.41.146:8302: i/o timeout
2021/05/07 11:34:14 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:34:15 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:46712->10.242.41.146:8300: i/o timeout
2021/05/07 11:34:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:17 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8302->10.242.41.146:42854: i/o timeout from=10.242.41.146:42854
2021/05/07 11:34:17 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8302->10.242.41.146:42854: i/o timeout from=10.242.41.146:42854
2021/05/07 11:34:17 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:34:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:21 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:34:21 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:34:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:58042->10.242.41.137:8301: i/o timeout
2021/05/07 11:34:22 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:34:23 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:46814->10.242.41.146:8300: i/o timeout
2021/05/07 11:34:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:24 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:34:24 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:34:24 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:34:26 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:34:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:58116->10.242.41.137:8301: i/o timeout
2021/05/07 11:34:26 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:34:26 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:34:26 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:34:26 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:34:27 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:46904->10.242.41.146:8300: i/o timeout
2021/05/07 11:34:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:33 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:34:33 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 11:34:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:35 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:34:35 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:34:36 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:47062->10.242.41.146:8300: i/o timeout
2021/05/07 11:34:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:42 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:34:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:43 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:47148->10.242.41.146:8300: i/o timeout
2021/05/07 11:34:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:44838->10.242.41.146:8302: i/o timeout
2021/05/07 11:34:44 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:34:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:48 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 11:34:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:49 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:34:49 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:34:49 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:34:49 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 11:34:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:50 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:34:51 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-x5pmr)
2021/05/07 11:34:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:53 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:34:53 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:34:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:34:55 [WARN] raft: AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300} rejected, sending older logs (next: 931)
2021/05/07 11:34:57 [INFO] raft: pipelining replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:34:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:58744->10.242.41.137:8301: i/o timeout
2021/05/07 11:34:59 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:35:01 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:35:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:03 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:35:03 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:35:03 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:35:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:58826->10.242.41.137:8301: i/o timeout
2021/05/07 11:35:04 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:35:05 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500140473s
2021/05/07 11:35:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:42438->10.242.41.146:8301: i/o timeout
2021/05/07 11:35:08 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:35:08 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:35:08 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:35:08 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:35:08 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:35:09 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:35:10 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:35:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:12 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:35:13 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:35:13 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:35:13 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:35:15 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:35:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:17 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500227938s
2021/05/07 11:35:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:42700->10.242.41.146:8301: i/o timeout
2021/05/07 11:35:18 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:35:19 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.993095234s
2021/05/07 11:35:19 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:35:19 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:35:19 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:35:19 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:35:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:22 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.451272992s
2021/05/07 11:35:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:24 [ERR] raft: Failed to pipeline AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: write tcp 10.242.139.204:47318->10.242.41.146:8300: use of closed network connection
2021/05/07 11:35:24 [INFO] raft: aborting pipeline replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:35:24 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:35:24 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:58292
2021/05/07 11:35:24 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:35:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:45656->10.242.41.146:8302: i/o timeout
2021/05/07 11:35:24 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:35:25 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:47584->10.242.41.146:8300: i/o timeout
2021/05/07 11:35:26 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:35:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:27 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.503777391s
2021/05/07 11:35:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:29 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:35:29 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:35:29 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:35:29 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:35:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:59336->10.242.41.137:8301: i/o timeout
2021/05/07 11:35:30 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:35:31 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:35:31 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:35:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:34 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500254551s
2021/05/07 11:35:34 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:48184->10.242.41.146:8300: i/o timeout
2021/05/07 11:35:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:42934->10.242.41.146:8301: i/o timeout
2021/05/07 11:35:34 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:35:35 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:35:35 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:35:36 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:35:36 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:35:36 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.94379804s
2021/05/07 11:35:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:37 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 11:35:38 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 6.574040229s
2021/05/07 11:35:38 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:35:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:41 [INFO] raft: pipelining replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:35:41 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:35:41 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:35:41 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:35:41 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:35:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:43120->10.242.41.146:8301: i/o timeout
2021/05/07 11:35:42 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:35:43 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:35:43 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500248416s
2021/05/07 11:35:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:43186->10.242.41.146:8301: i/o timeout
2021/05/07 11:35:45 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:35:46 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.931844322s
2021/05/07 11:35:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:46 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:35:46 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:35:46 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:35:48 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.317066435s
2021/05/07 11:35:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:51 [INFO] raft: aborting pipeline replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:35:51 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:35:52 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:48222->10.242.41.146:8300: i/o timeout
2021/05/07 11:35:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:53 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:35:53 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500198888s
2021/05/07 11:35:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:46172->10.242.41.146:8302: i/o timeout
2021/05/07 11:35:54 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:35:56 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.937562222s
2021/05/07 11:35:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:35:58 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.37551484s
2021/05/07 11:35:59 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:35:59 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/07 11:35:59 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 11:36:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:36:01 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:48640->10.242.41.146:8300: i/o timeout
2021/05/07 11:36:01 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:36:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:36:03 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:48660->10.242.41.146:8300: i/o timeout
2021/05/07 11:36:03 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:36:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:36:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:36:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:36:09 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 11:36:09 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 11:36:09 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:36:09 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:36:09 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:36:09 [ERR] raft: peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300} has newer term, stopping replication
2021/05/07 11:36:09 [INFO] raft: Node at 10.242.139.204:8300 [Follower] entering Follower state (Leader: "")
2021/05/07 11:36:09 [INFO] raft: aborting pipeline replication to peer {Voter dff60f1e-22b9-7f6a-153b-e2e4aea758bb 10.242.113.198:8300}
2021/05/07 11:36:09 [ERR] consul: failed to reconcile member: {dev-consul-server-1 10.242.41.146 8301 map[id:e90a9b84-7b02-fc1f-15b4-2419fe66fc38 vsn_min:2 dc:dc1 expect:3 vsn:2 segment: vsn_max:3 build:1.0.6:9a494b5f wan_join_port:8302 role:consul raft_vsn:3 port:8300] alive 1 5 2 2 5 4}: leadership lost while committing log
2021/05/07 11:36:09 [INFO] consul: cluster leadership lost
2021/05/07 11:36:09 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:36:11 [WARN] consul.coordinate: Batch update failed: node is not the leader
2021/05/07 11:36:16 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/05/07 11:36:16 [INFO] raft: Node at 10.242.139.204:8300 [Candidate] entering Candidate state in term 7
2021/05/07 11:36:16 [INFO] raft: Node at 10.242.139.204:8300 [Follower] entering Follower state (Leader: "")
2021/05/07 11:36:16 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since our last index is greater (1147, 1132)
2021/05/07 11:36:17 [ERR] http: Request GET /v1/catalog/service/holmes-rule-mgmt, error: No cluster leader from=10.242.41.155:59532
2021/05/07 11:36:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:60214->10.242.41.137:8301: i/o timeout
2021/05/07 11:36:21 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:36:25 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:36:25 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:36:25 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/05/07 11:36:25 [INFO] raft: Node at 10.242.139.204:8300 [Candidate] entering Candidate state in term 9
2021/05/07 11:36:25 [INFO] raft: Election won. Tally: 2
2021/05/07 11:36:25 [INFO] raft: Node at 10.242.139.204:8300 [Leader] entering Leader state
2021/05/07 11:36:25 [INFO] raft: Added peer dff60f1e-22b9-7f6a-153b-e2e4aea758bb, starting replication
2021/05/07 11:36:25 [INFO] raft: Added peer e90a9b84-7b02-fc1f-15b4-2419fe66fc38, starting replication
2021/05/07 11:36:25 [INFO] consul: cluster leadership acquired
2021/05/07 11:36:25 [INFO] consul: New leader elected: dev-consul-server-2
2021/05/07 11:36:25 [INFO] raft: pipelining replication to peer {Voter dff60f1e-22b9-7f6a-153b-e2e4aea758bb 10.242.113.198:8300}
2021/05/07 11:36:25 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:36:25 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:36:25 [WARN] raft: AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300} rejected, sending older logs (next: 1133)
2021/05/07 11:36:26 [INFO] raft: pipelining replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:36:34 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:36:34 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 11:36:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:36:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:36:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:60512->10.242.41.137:8301: i/o timeout
2021/05/07 11:36:38 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:36:48 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:36:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:60694->10.242.41.137:8301: i/o timeout
2021/05/07 11:36:48 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:36:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:36:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:36:52 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500236353s
2021/05/07 11:36:52 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-x5pmr but other probes failed, network may be misconfigured
2021/05/07 11:36:53 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:36:53 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:36:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:36:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:47218->10.242.41.146:8302: i/o timeout
2021/05/07 11:36:54 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:36:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:44366->10.242.41.146:8301: i/o timeout
2021/05/07 11:36:54 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:36:55 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.909085042s
2021/05/07 11:36:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:36:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:36:57 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:36:57 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:36:57 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:36:57 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.397792508s
2021/05/07 11:36:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:44422->10.242.41.146:8301: i/o timeout
2021/05/07 11:36:57 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:36:59 [INFO] raft: aborting pipeline replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:36:59 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:36:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:37:00 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:48814->10.242.41.146:8300: i/o timeout
2021/05/07 11:37:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:37:01 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500161445s
2021/05/07 11:37:01 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:37:04 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.945825553s
2021/05/07 11:37:04 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:37:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:47312->10.242.41.146:8302: i/o timeout
2021/05/07 11:37:04 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:37:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:37:06 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.403969301s
2021/05/07 11:37:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:37:09 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:49818->10.242.41.146:8300: i/o timeout
2021/05/07 11:37:09 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:37:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:37:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:37:11 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:49864->10.242.41.146:8300: i/o timeout
2021/05/07 11:37:12 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:37:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:37:13 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:37:13 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:37:13 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:37:14 [INFO] raft: pipelining replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:37:14 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:37:14 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:37:26 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:37:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/07 11:37:26 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 11:37:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:33152->10.242.41.137:8301: i/o timeout
2021/05/07 11:37:26 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:37:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:44978->10.242.41.146:8301: i/o timeout
2021/05/07 11:37:30 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500176234s
2021/05/07 11:37:30 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:37:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:33214->10.242.41.137:8301: i/o timeout
2021/05/07 11:37:30 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:37:30 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:37:30 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:37:30 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:37:32 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:37:32 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:37:34 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:37:34 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/07 11:37:34 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 11:37:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:37:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:45118->10.242.41.146:8301: i/o timeout
2021/05/07 11:37:35 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:37:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:37:37 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500330332s
2021/05/07 11:37:38 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:37:38 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:37:38 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:37:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:33338->10.242.41.137:8301: i/o timeout
2021/05/07 11:37:38 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:37:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:37:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:45202->10.242.41.146:8301: i/o timeout
2021/05/07 11:37:40 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:37:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:37:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:37:43 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.503259608s
2021/05/07 11:37:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:48132->10.242.41.146:8302: i/o timeout
2021/05/07 11:37:44 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:37:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:45280->10.242.41.146:8301: i/o timeout
2021/05/07 11:37:44 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:37:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:37:46 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.989639572s
2021/05/07 11:37:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:37:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:45342->10.242.41.146:8301: i/o timeout
2021/05/07 11:37:47 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:37:48 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.47005188s
2021/05/07 11:37:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:37:50 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:50048->10.242.41.146:8300: i/o timeout
2021/05/07 11:37:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:45372->10.242.41.146:8301: i/o timeout
2021/05/07 11:37:50 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:37:51 [ERR] raft: Failed to pipeline AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: write tcp 10.242.139.204:49992->10.242.41.146:8300: use of closed network connection
2021/05/07 11:37:51 [INFO] raft: aborting pipeline replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:37:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:37:52 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations)
2021/05/07 11:37:52 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:37:52 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:37:52 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:37:53 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:37:53 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:37:53 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.501139031s
2021/05/07 11:37:54 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:37:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:37:56 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.988829028s
2021/05/07 11:37:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:37:57 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:37:57 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:59956
2021/05/07 11:37:58 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.433887468s
2021/05/07 11:37:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:37:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:48404->10.242.41.146:8302: i/o timeout
2021/05/07 11:37:59 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:38:00 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:38:00 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:50760->10.242.41.146:8300: i/o timeout
2021/05/07 11:38:01 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:38:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:01 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:50776->10.242.41.146:8300: i/o timeout
2021/05/07 11:38:01 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:38:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:09 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8301->10.242.41.137:41368: i/o timeout from=10.242.41.137:41368
2021/05/07 11:38:09 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8301->10.242.41.137:41368: i/o timeout from=10.242.41.137:41368
2021/05/07 11:38:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:48586->10.242.41.146:8302: i/o timeout
2021/05/07 11:38:09 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:38:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:11 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:50952->10.242.41.146:8300: i/o timeout
2021/05/07 11:38:11 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:50954->10.242.41.146:8300: i/o timeout
2021/05/07 11:38:11 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:38:12 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:38:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:14 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:38:14 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:38:14 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:38:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:17 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:38:17 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.146:55434
2021/05/07 11:38:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:21 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:51128->10.242.41.146:8300: i/o timeout
2021/05/07 11:38:21 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:38:22 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:51144->10.242.41.146:8300: i/o timeout
2021/05/07 11:38:22 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:38:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:48762->10.242.41.146:8302: i/o timeout
2021/05/07 11:38:24 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:38:24 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:38:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:31 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:51312->10.242.41.146:8300: i/o timeout
2021/05/07 11:38:31 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:38:32 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:38:32 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:38:32 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:51332->10.242.41.146:8300: i/o timeout
2021/05/07 11:38:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:33 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:38:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:49142->10.242.41.146:8302: i/o timeout
2021/05/07 11:38:39 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:38:40 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:38:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:41 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:51500->10.242.41.146:8300: i/o timeout
2021/05/07 11:38:41 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:38:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:43 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:51546->10.242.41.146:8300: i/o timeout
2021/05/07 11:38:44 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:38:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:49242->10.242.41.146:8302: i/o timeout
2021/05/07 11:38:49 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:38:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:51 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:51706->10.242.41.146:8300: i/o timeout
2021/05/07 11:38:51 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:38:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:54 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:51790->10.242.41.146:8300: i/o timeout
2021/05/07 11:38:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:55 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:38:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:38:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:01 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:51928->10.242.41.146:8300: i/o timeout
2021/05/07 11:39:02 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:39:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:49636->10.242.41.146:8302: i/o timeout
2021/05/07 11:39:04 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:39:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:05 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:52002->10.242.41.146:8300: i/o timeout
2021/05/07 11:39:06 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:39:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:09 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:39:09 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:39:09 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:39:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:12 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:52112->10.242.41.146:8300: i/o timeout
2021/05/07 11:39:12 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:39:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:49810->10.242.41.146:8302: i/o timeout
2021/05/07 11:39:14 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:39:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:16 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:52202->10.242.41.146:8300: i/o timeout
2021/05/07 11:39:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:19 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:39:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:20 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:39:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:22 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:52296->10.242.41.146:8300: i/o timeout
2021/05/07 11:39:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:24 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:39:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:29 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:52412->10.242.41.146:8300: i/o timeout
2021/05/07 11:39:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:32 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:39:32 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:39:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:34 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:52506->10.242.41.146:8300: i/o timeout
2021/05/07 11:39:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:36 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:39:36 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:39:36 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:39:38 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:39:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:42 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:52634->10.242.41.146:8300: i/o timeout
2021/05/07 11:39:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:46 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:52712->10.242.41.146:8300: i/o timeout
2021/05/07 11:39:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:48 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:39:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:51 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:39:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:39:58 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:52918->10.242.41.146:8300: i/o timeout
2021/05/07 11:39:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:00 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:40:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:01 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:52972->10.242.41.146:8300: i/o timeout
2021/05/07 11:40:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:09 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:40:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:12 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:40:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:17 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:40:17 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:40:19 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 11:40:19 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:53276->10.242.41.146:8300: i/o timeout
2021/05/07 11:40:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:40:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:22 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:53320->10.242.41.146:8300: i/o timeout
2021/05/07 11:40:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:30 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:40:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:32 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:40:32 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:40:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:34 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:40:34 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:40:34 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:40:34 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:40:34 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:40:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:48376->10.242.41.146:8301: i/o timeout
2021/05/07 11:40:35 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:40:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:39 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500172174s
2021/05/07 11:40:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:48440->10.242.41.146:8301: i/o timeout
2021/05/07 11:40:39 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:40:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:41 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.978537371s
2021/05/07 11:40:41 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:40:41 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:40:41 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:40:41 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:40:42 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:53670->10.242.41.146:8300: i/o timeout
2021/05/07 11:40:42 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:40:42 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:40:42 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/05/07 11:40:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:44 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.415481139s
2021/05/07 11:40:44 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 11:40:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:36698->10.242.41.137:8301: i/o timeout
2021/05/07 11:40:44 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:40:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:47 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:53642->10.242.41.146:8300: i/o timeout
2021/05/07 11:40:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:36780->10.242.41.137:8301: i/o timeout
2021/05/07 11:40:47 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:40:48 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:40:48 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:40:48 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:40:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:52 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:40:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:51560->10.242.41.146:8302: i/o timeout
2021/05/07 11:40:54 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:40:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:58 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:53982->10.242.41.146:8300: i/o timeout
2021/05/07 11:40:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:40:59 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:40:59 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:40:59 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:41:02 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:41:02 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:54038->10.242.41.146:8300: i/o timeout
2021/05/07 11:41:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:51730->10.242.41.146:8302: i/o timeout
2021/05/07 11:41:04 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:41:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:08 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:41:08 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.146:59286
2021/05/07 11:41:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:09 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:54160->10.242.41.146:8300: i/o timeout
2021/05/07 11:41:10 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:41:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:12 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:41:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:52000->10.242.41.146:8302: i/o timeout
2021/05/07 11:41:19 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:41:20 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:54350->10.242.41.146:8300: i/o timeout
2021/05/07 11:41:21 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:41:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:22 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:54388->10.242.41.146:8300: i/o timeout
2021/05/07 11:41:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:24 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:41:24 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:41:24 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:41:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:52084->10.242.41.146:8302: i/o timeout
2021/05/07 11:41:29 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:41:31 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:54542->10.242.41.146:8300: i/o timeout
2021/05/07 11:41:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:31 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:41:33 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:41:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:34 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:41:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:40 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:41:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:41 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:41:41 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:41:41 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:54730->10.242.41.146:8300: i/o timeout
2021/05/07 11:41:43 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:54744->10.242.41.146:8300: i/o timeout
2021/05/07 11:41:43 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:41:44 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:41:44 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/07 11:41:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:53 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:54920->10.242.41.146:8300: i/o timeout
2021/05/07 11:41:53 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:41:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:54 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:41:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:41:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:03 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:55100->10.242.41.146:8300: i/o timeout
2021/05/07 11:42:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:04 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:55118->10.242.41.146:8300: i/o timeout
2021/05/07 11:42:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:05 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:42:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:13 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:42:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:15 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:55318->10.242.41.146:8300: i/o timeout
2021/05/07 11:42:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:17 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:42:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:20 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:42:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:23 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:42:23 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:42:23 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:55444->10.242.41.146:8300: i/o timeout
2021/05/07 11:42:26 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:42:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:27 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:55504->10.242.41.146:8300: i/o timeout
2021/05/07 11:42:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:30 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:42:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:33 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:42:34 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:42:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:35 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:42:35 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.146:60230
2021/05/07 11:42:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:37 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8301->10.242.41.137:43416: i/o timeout from=10.242.41.137:43416
2021/05/07 11:42:37 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8301->10.242.41.137:43416: i/o timeout from=10.242.41.137:43416
2021/05/07 11:42:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:40 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:55742->10.242.41.146:8300: i/o timeout
2021/05/07 11:42:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:43 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:55796->10.242.41.146:8300: i/o timeout
2021/05/07 11:42:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:46 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:42:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:54 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:42:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:56 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:56026->10.242.41.146:8300: i/o timeout
2021/05/07 11:42:57 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8302->10.242.41.146:48604: i/o timeout from=10.242.41.146:48604
2021/05/07 11:42:57 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8302->10.242.41.146:48604: i/o timeout from=10.242.41.146:48604
2021/05/07 11:42:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:42:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:00 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:43:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:04 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:56150->10.242.41.146:8300: i/o timeout
2021/05/07 11:43:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:05 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:43:05 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:43:06 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:43:06 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:35142
2021/05/07 11:43:07 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:43:08 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:43:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:14 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:43:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:16 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:43:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:17 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:43:17 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:43:18 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500147021s
2021/05/07 11:43:18 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:43:18 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:43:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:39420->10.242.41.137:8301: i/o timeout
2021/05/07 11:43:19 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:43:20 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.998999515s
2021/05/07 11:43:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:22 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:43:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:39458->10.242.41.137:8301: i/o timeout
2021/05/07 11:43:22 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:43:23 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/07 11:43:23 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:43:23 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:43:23 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:43:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:24 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:56490->10.242.41.146:8300: i/o timeout
2021/05/07 11:43:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:39554->10.242.41.137:8301: i/o timeout
2021/05/07 11:43:26 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:43:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:27 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500206544s
2021/05/07 11:43:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:30 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.94102102s
2021/05/07 11:43:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:39600->10.242.41.137:8301: i/o timeout
2021/05/07 11:43:30 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:43:30 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:43:30 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:43:30 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:43:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:32 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.409979561s
2021/05/07 11:43:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:51454->10.242.41.146:8301: i/o timeout
2021/05/07 11:43:32 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:43:32 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:43:32 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:43:32 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:43:32 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:43:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:34 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:43:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:35 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:56378->10.242.41.146:8300: i/o timeout
2021/05/07 11:43:36 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:43:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:40 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:43:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:41 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:43:41 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:43:41 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:43:44 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:43:44 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:56858->10.242.41.146:8300: i/o timeout
2021/05/07 11:43:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:45 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500254617s
2021/05/07 11:43:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:51722->10.242.41.146:8301: i/o timeout
2021/05/07 11:43:46 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:43:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:49 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:43:49 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:43:49 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:43:49 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:43:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:49 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500132714s
2021/05/07 11:43:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:51754->10.242.41.146:8301: i/o timeout
2021/05/07 11:43:49 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:43:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:52 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.995839094s
2021/05/07 11:43:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:54 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.469477204s
2021/05/07 11:43:54 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:43:54 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 11:43:55 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:43:55 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:43:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:57 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-6xm62)
2021/05/07 11:43:57 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:56900->10.242.41.146:8300: i/o timeout
2021/05/07 11:43:58 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:43:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:43:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:54796->10.242.41.146:8302: i/o timeout
2021/05/07 11:43:59 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:43:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:40146->10.242.41.137:8301: i/o timeout
2021/05/07 11:43:59 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:44:00 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:44:00 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:44:00 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:44:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:04 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:44:04 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 11:44:04 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:57230->10.242.41.146:8300: i/o timeout
2021/05/07 11:44:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:40234->10.242.41.137:8301: i/o timeout
2021/05/07 11:44:04 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:44:05 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500880452s
2021/05/07 11:44:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:40318->10.242.41.137:8301: i/o timeout
2021/05/07 11:44:08 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:44:08 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:44:08 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:44:08 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:44:09 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.50027543s
2021/05/07 11:44:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:11 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.985159641s
2021/05/07 11:44:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:52142->10.242.41.146:8301: i/o timeout
2021/05/07 11:44:12 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:44:13 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:44:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:13 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:44:13 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:44:13 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:44:14 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.48802158s
2021/05/07 11:44:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:55072->10.242.41.146:8302: i/o timeout
2021/05/07 11:44:14 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:44:15 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:44:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:17 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:57298->10.242.41.146:8300: i/o timeout
2021/05/07 11:44:18 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:44:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:55176->10.242.41.146:8302: i/o timeout
2021/05/07 11:44:24 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:44:25 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:57618->10.242.41.146:8300: i/o timeout
2021/05/07 11:44:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:28 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:57672->10.242.41.146:8300: i/o timeout
2021/05/07 11:44:28 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:44:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:33 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:44:33 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.146:60542
2021/05/07 11:44:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:55432->10.242.41.146:8302: i/o timeout
2021/05/07 11:44:34 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:44:35 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:44:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:38 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:57864->10.242.41.146:8300: i/o timeout
2021/05/07 11:44:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:39 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:44:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:41 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8301->10.242.41.137:44432: i/o timeout from=10.242.41.137:44432
2021/05/07 11:44:41 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8301->10.242.41.137:44432: i/o timeout from=10.242.41.137:44432
2021/05/07 11:44:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:44 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:44:44 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:44:44 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:44:45 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:57976->10.242.41.146:8300: i/o timeout
2021/05/07 11:44:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:47 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:44:47 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:44:47 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:44:47 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:44:47 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:44:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:53 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:44:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:55 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:44:55 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:44:55 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-x5pmr)
2021/05/07 11:44:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:44:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:41198->10.242.41.137:8301: i/o timeout
2021/05/07 11:44:57 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:44:57 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500232396s
2021/05/07 11:44:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:01 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:45:01 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:45:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:53066->10.242.41.146:8301: i/o timeout
2021/05/07 11:45:01 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:45:01 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.50023252s
2021/05/07 11:45:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:53112->10.242.41.146:8301: i/o timeout
2021/05/07 11:45:03 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:45:04 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.993170654s
2021/05/07 11:45:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:55980->10.242.41.146:8302: i/o timeout
2021/05/07 11:45:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:05 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:58054->10.242.41.146:8300: i/o timeout
2021/05/07 11:45:05 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:45:05 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:45:05 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:45:05 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:45:07 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.50026439s
2021/05/07 11:45:08 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:45:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:10 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.953985597s
2021/05/07 11:45:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:12 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.434256328s
2021/05/07 11:45:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:15 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:58352->10.242.41.146:8300: i/o timeout
2021/05/07 11:45:15 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:45:16 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:45:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:56336->10.242.41.146:8302: i/o timeout
2021/05/07 11:45:24 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:45:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:25 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:58700->10.242.41.146:8300: i/o timeout
2021/05/07 11:45:26 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:58718->10.242.41.146:8300: i/o timeout
2021/05/07 11:45:26 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:45:26 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:36546
2021/05/07 11:45:27 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:45:27 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:45:27 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:45:28 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:45:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:33 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:45:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:56444->10.242.41.146:8302: i/o timeout
2021/05/07 11:45:34 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:45:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:35 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:45:37 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:58910->10.242.41.146:8300: i/o timeout
2021/05/07 11:45:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:37 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:45:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:41 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:45:41 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:45:41 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:45:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:45 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:59056->10.242.41.146:8300: i/o timeout
2021/05/07 11:45:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:47 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:59092->10.242.41.146:8300: i/o timeout
2021/05/07 11:45:48 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:45:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:56 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:45:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:58 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:59290->10.242.41.146:8300: i/o timeout
2021/05/07 11:45:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:45:59 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:46:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:06 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:59408->10.242.41.146:8300: i/o timeout
2021/05/07 11:46:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:07 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:46:07 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:46:07 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:46:07 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.146:33486
2021/05/07 11:46:09 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:59482->10.242.41.146:8300: i/o timeout
2021/05/07 11:46:10 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:46:10 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:46:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:13 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:46:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:16 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:46:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:20 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:59658->10.242.41.146:8300: i/o timeout
2021/05/07 11:46:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:21 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:46:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:26 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:59766->10.242.41.146:8300: i/o timeout
2021/05/07 11:46:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:31 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:59866->10.242.41.146:8300: i/o timeout
2021/05/07 11:46:33 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:46:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:34 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:46:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:36 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:46:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:43 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:60044->10.242.41.146:8300: i/o timeout
2021/05/07 11:46:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:45 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:46:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:46 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:60122->10.242.41.146:8300: i/o timeout
2021/05/07 11:46:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:49 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:46:49 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:46:52 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:46:53 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:46:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:55 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:60290->10.242.41.146:8300: i/o timeout
2021/05/07 11:46:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:56 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:46:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:46:58 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:46:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:47:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:47:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:47:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:47:06 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:60478->10.242.41.146:8300: i/o timeout
2021/05/07 11:47:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:47:08 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:60512->10.242.41.146:8300: i/o timeout
2021/05/07 11:47:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:47:10 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:47:10 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:47:10 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:47:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:47:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:55384->10.242.41.146:8301: i/o timeout
2021/05/07 11:47:12 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:47:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:47:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:47:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:55420->10.242.41.146:8301: i/o timeout
2021/05/07 11:47:15 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:47:16 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:47:16 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:47:16 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:47:16 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:47:16 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:47:16 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:47:16 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:47:17 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:47:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:47:17 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:47:17 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:47:17 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:47:19 [INFO] raft: pipelining replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:47:21 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:47:21 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:47:30 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:47:34 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:47:34 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/07 11:47:38 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:47:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:47:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:55906->10.242.41.146:8301: i/o timeout
2021/05/07 11:47:40 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:47:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:47:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:47:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:44116->10.242.41.137:8301: i/o timeout
2021/05/07 11:47:43 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:47:43 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:47:43 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:47:43 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:47:44 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:47:44 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:47:44 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:47:44 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:47:45 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.584736237s
2021/05/07 11:47:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:47:46 [INFO] raft: aborting pipeline replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:47:46 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:47:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:47:48 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:47:50 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:47:50 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:47:50 [INFO] raft: pipelining replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:47:50 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:47:50 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:47:50 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:47:50 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:47:50 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:47:52 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:47:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:56124->10.242.41.146:8301: i/o timeout
2021/05/07 11:47:52 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:47:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:47:54 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500268651s
2021/05/07 11:47:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:47:56 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.954407852s
2021/05/07 11:47:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:44396->10.242.41.137:8301: i/o timeout
2021/05/07 11:47:56 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:47:57 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:47:57 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:47:57 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:47:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:47:58 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:47:58 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:47:59 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.438608401s
2021/05/07 11:47:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:47:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:59108->10.242.41.146:8302: i/o timeout
2021/05/07 11:47:59 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:47:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:56218->10.242.41.146:8301: i/o timeout
2021/05/07 11:47:59 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:48:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:48:01 [INFO] raft: aborting pipeline replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:48:01 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:48:02 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:60778->10.242.41.146:8300: i/o timeout
2021/05/07 11:48:03 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:48:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:48:04 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500261732s
2021/05/07 11:48:04 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:48:04 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:48:04 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:48:04 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/07 11:48:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:48:06 [INFO] raft: pipelining replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:48:06 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:48:06 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:48:16 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:48:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:48:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:56580->10.242.41.146:8301: i/o timeout
2021/05/07 11:48:17 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:48:18 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500955182s
2021/05/07 11:48:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:48:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:44798->10.242.41.137:8301: i/o timeout
2021/05/07 11:48:20 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:48:20 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:48:20 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:48:20 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:48:20 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:48:21 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:48:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:48:22 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:48:23 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:48:23 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:48:23 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:48:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:48:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:44876->10.242.41.137:8301: i/o timeout
2021/05/07 11:48:23 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:48:23 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:48:24 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:48:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:59572->10.242.41.146:8302: i/o timeout
2021/05/07 11:48:24 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:48:25 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:48:25 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:48:25 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:48:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:48:25 [ERR] raft: Failed to pipeline AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: write tcp 10.242.139.204:33432->10.242.41.146:8300: use of closed network connection
2021/05/07 11:48:25 [INFO] raft: aborting pipeline replication to peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}
2021/05/07 11:48:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:48:27 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:48:27 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.146:34490
2021/05/07 11:48:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:56780->10.242.41.146:8301: i/o timeout
2021/05/07 11:48:27 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:48:29 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500321776s
2021/05/07 11:48:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:48:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:56826->10.242.41.146:8301: i/o timeout
2021/05/07 11:48:30 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:48:30 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:48:30 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:48:30 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:48:30 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:48:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:48:31 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.953033595s
2021/05/07 11:48:33 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:48:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:48:33 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 11:48:34 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.431115906s
2021/05/07 11:48:34 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 11:48:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:48:35 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:33888->10.242.41.146:8300: i/o timeout
2021/05/07 11:48:35 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:48:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:48:37 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:33458->10.242.41.146:8300: i/o timeout
2021/05/07 11:48:38 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:48:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:48:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:48:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:48:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:59930->10.242.41.146:8302: i/o timeout
2021/05/07 11:48:44 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:48:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:48:45 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:34056->10.242.41.146:8300: i/o timeout
2021/05/07 11:48:45 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:48:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:48:48 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:34096->10.242.41.146:8300: i/o timeout
2021/05/07 11:48:49 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:48:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:48:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:48:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:48:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:60110->10.242.41.146:8302: i/o timeout
2021/05/07 11:48:54 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:48:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:48:55 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:34250->10.242.41.146:8300: i/o timeout
2021/05/07 11:48:55 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:48:56 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:48:57 [ERR] raft: peer {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300} has newer term, stopping replication
2021/05/07 11:48:57 [INFO] raft: Node at 10.242.139.204:8300 [Follower] entering Follower state (Leader: "")
2021/05/07 11:48:57 [INFO] raft: aborting pipeline replication to peer {Voter dff60f1e-22b9-7f6a-153b-e2e4aea758bb 10.242.113.198:8300}
2021/05/07 11:48:57 [INFO] consul: cluster leadership lost
2021/05/07 11:49:00 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 11:49:00 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:49:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:45582->10.242.41.137:8301: i/o timeout
2021/05/07 11:49:01 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:49:04 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:49:04 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:49:04 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.139.203:37394
2021/05/07 11:49:04 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 11:49:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:45616->10.242.41.137:8301: i/o timeout
2021/05/07 11:49:04 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:49:04 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/05/07 11:49:04 [INFO] raft: Node at 10.242.139.204:8300 [Candidate] entering Candidate state in term 10
2021/05/07 11:49:05 [INFO] raft: Election won. Tally: 2
2021/05/07 11:49:05 [INFO] raft: Node at 10.242.139.204:8300 [Leader] entering Leader state
2021/05/07 11:49:05 [INFO] raft: Added peer dff60f1e-22b9-7f6a-153b-e2e4aea758bb, starting replication
2021/05/07 11:49:05 [INFO] raft: Added peer e90a9b84-7b02-fc1f-15b4-2419fe66fc38, starting replication
2021/05/07 11:49:05 [INFO] consul: cluster leadership acquired
2021/05/07 11:49:05 [INFO] consul: New leader elected: dev-consul-server-2
2021/05/07 11:49:05 [INFO] raft: pipelining replication to peer {Voter dff60f1e-22b9-7f6a-153b-e2e4aea758bb 10.242.113.198:8300}
2021/05/07 11:49:05 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:49:05 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:49:06 [INFO] memberlist: Marking dev-consul-7467bf7b46-x5pmr as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:49:06 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:49:06 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:49:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:57478->10.242.41.146:8301: i/o timeout
2021/05/07 11:49:06 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:49:07 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.501739535s
2021/05/07 11:49:08 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:49:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:57540->10.242.41.146:8301: i/o timeout
2021/05/07 11:49:09 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:49:09 [ERR] yamux: keepalive failed: session shutdown
2021/05/07 11:49:09 [ERR] consul.rpc: multiplex conn accept failed: read tcp 10.242.139.204:8300->10.242.41.137:37726: read: connection reset by peer from=10.242.41.137:37726
2021/05/07 11:49:10 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.953908787s
2021/05/07 11:49:10 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:49:10 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:49:10 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:49:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:12 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.439413045s
2021/05/07 11:49:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:57574->10.242.41.146:8301: i/o timeout
2021/05/07 11:49:12 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:49:13 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:49:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:14 [ERR] raft: Failed to make RequestVote RPC to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:34426->10.242.41.146:8300: i/o timeout
2021/05/07 11:49:15 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:34304->10.242.41.146:8300: i/o timeout
2021/05/07 11:49:15 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:49:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:15 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:34612->10.242.41.146:8300: i/o timeout
2021/05/07 11:49:16 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:49:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:60578->10.242.41.146:8302: i/o timeout
2021/05/07 11:49:19 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:49:20 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:49:20 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:49:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:25 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:34784->10.242.41.146:8300: i/o timeout
2021/05/07 11:49:25 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:49:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:26 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:34806->10.242.41.146:8300: i/o timeout
2021/05/07 11:49:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:27 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:49:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:60672->10.242.41.146:8302: i/o timeout
2021/05/07 11:49:29 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:49:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:35 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:34980->10.242.41.146:8300: i/o timeout
2021/05/07 11:49:35 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:49:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:37 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:35008->10.242.41.146:8300: i/o timeout
2021/05/07 11:49:38 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:49:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:45 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:35144->10.242.41.146:8300: i/o timeout
2021/05/07 11:49:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:45 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:49:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:48 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:35188->10.242.41.146:8300: i/o timeout
2021/05/07 11:49:49 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:49:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:55 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:35330->10.242.41.146:8300: i/o timeout
2021/05/07 11:49:55 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:49:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:59 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:35390->10.242.41.146:8300: i/o timeout
2021/05/07 11:49:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:49:59 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:50:00 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 11:50:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:05 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:35506->10.242.41.146:8300: i/o timeout
2021/05/07 11:50:05 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:50:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:09 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:35582->10.242.41.146:8300: i/o timeout
2021/05/07 11:50:10 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:50:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:15 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:35684->10.242.41.146:8300: i/o timeout
2021/05/07 11:50:15 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:50:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:18 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:50:18 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:50:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:50:20 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 11:50:20 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:35766->10.242.41.146:8300: i/o timeout
2021/05/07 11:50:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:22 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:50:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:25 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:35852->10.242.41.146:8300: i/o timeout
2021/05/07 11:50:26 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:50:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:32 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:35982->10.242.41.146:8300: i/o timeout
2021/05/07 11:50:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:33 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:50:34 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:50:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:36 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:36062->10.242.41.146:8300: i/o timeout
2021/05/07 11:50:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:37 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:50:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:43 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:36170->10.242.41.146:8300: i/o timeout
2021/05/07 11:50:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:45 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:50:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:47 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:36240->10.242.41.146:8300: i/o timeout
2021/05/07 11:50:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:50 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:50:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:53 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:50:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:55 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:36392->10.242.41.146:8300: i/o timeout
2021/05/07 11:50:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:58 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:50:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:50:59 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:50:59 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:51:00 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:36466->10.242.41.146:8300: i/o timeout
2021/05/07 11:51:02 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:51:02 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/07 11:51:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:05 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:51:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:08 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:36618->10.242.41.146:8300: i/o timeout
2021/05/07 11:51:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:14 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:51:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:15 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:36748->10.242.41.146:8300: i/o timeout
2021/05/07 11:51:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:24 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:36906->10.242.41.146:8300: i/o timeout
2021/05/07 11:51:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:25 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:51:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:35 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:51:35 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:37104->10.242.41.146:8300: i/o timeout
2021/05/07 11:51:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:39 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:51:39 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:51:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:41 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:51:41 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:51:44 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:51:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:34904->10.242.41.146:8302: i/o timeout
2021/05/07 11:51:44 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:51:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:45 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:37262->10.242.41.146:8300: i/o timeout
2021/05/07 11:51:45 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:51:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:35084->10.242.41.146:8302: i/o timeout
2021/05/07 11:51:54 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:51:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:55 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:37454->10.242.41.146:8300: i/o timeout
2021/05/07 11:51:56 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:51:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:51:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:35262->10.242.41.146:8302: i/o timeout
2021/05/07 11:52:04 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:52:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:06 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:52:06 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:37636->10.242.41.146:8300: i/o timeout
2021/05/07 11:52:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:09 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8302->10.242.41.146:53632: i/o timeout from=10.242.41.146:53632
2021/05/07 11:52:09 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8302->10.242.41.146:53632: i/o timeout from=10.242.41.146:53632
2021/05/07 11:52:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:35354->10.242.41.146:8302: i/o timeout
2021/05/07 11:52:14 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:52:14 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:52:14 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:52:14 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:52:15 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:52:15 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.146:37172
2021/05/07 11:52:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:16 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:37808->10.242.41.146:8300: i/o timeout
2021/05/07 11:52:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:17 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:52:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:23 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:52:23 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:52:26 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:52:26 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:52:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:27 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:37992->10.242.41.146:8300: i/o timeout
2021/05/07 11:52:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:33 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:52:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:36 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:38176->10.242.41.146:8300: i/o timeout
2021/05/07 11:52:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:38 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:52:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:46 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:52:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:48 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:38374->10.242.41.146:8300: i/o timeout
2021/05/07 11:52:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:56 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:38528->10.242.41.146:8300: i/o timeout
2021/05/07 11:52:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:52:59 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:53:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:05 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:53:05 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:53:06 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:53:08 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:53:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/07 11:53:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:09 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:38744->10.242.41.146:8300: i/o timeout
2021/05/07 11:53:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:13 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:53:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:16 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:38870->10.242.41.146:8300: i/o timeout
2021/05/07 11:53:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:19 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:53:19 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' joined, marking health alive
2021/05/07 11:53:20 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:53:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:50194->10.242.41.137:8301: i/o timeout
2021/05/07 11:53:21 [INFO] memberlist: Suspect dev-consul-7467bf7b46-x5pmr has failed, no acks received
2021/05/07 11:53:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:23 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:53:23 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:53:23 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/05/07 11:53:23 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 11:53:23 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:53:23 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 11:53:25 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-x5pmr 10.242.41.137
2021/05/07 11:53:25 [INFO] consul: member 'dev-consul-7467bf7b46-x5pmr' failed, marking health critical
2021/05/07 11:53:26 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:53:26 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/07 11:53:26 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 11:53:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:33864->10.242.41.146:8301: i/o timeout
2021/05/07 11:53:26 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:53:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:28 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500185305s
2021/05/07 11:53:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:33882->10.242.41.146:8301: i/o timeout
2021/05/07 11:53:28 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:53:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:30 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.95825944s
2021/05/07 11:53:30 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 11:53:30 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 11:53:30 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 11:53:30 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 11:53:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:33950->10.242.41.146:8301: i/o timeout
2021/05/07 11:53:31 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 11:53:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:33 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:53:34 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500153312s
2021/05/07 11:53:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:37 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:39240->10.242.41.146:8300: i/o timeout
2021/05/07 11:53:37 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.933817319s
2021/05/07 11:53:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:39 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.375401732s
2021/05/07 11:53:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:36932->10.242.41.146:8302: i/o timeout
2021/05/07 11:53:39 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:53:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:43 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:39114->10.242.41.146:8300: i/o timeout
2021/05/07 11:53:43 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:53:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:47 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:53:47 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:53:47 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:53:49 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:53:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:37026->10.242.41.146:8302: i/o timeout
2021/05/07 11:53:49 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:53:50 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:53:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:53 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:39534->10.242.41.146:8300: i/o timeout
2021/05/07 11:53:54 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:53:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:57 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:39594->10.242.41.146:8300: i/o timeout
2021/05/07 11:53:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:53:59 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:53:59 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:40848
2021/05/07 11:54:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:37296->10.242.41.146:8302: i/o timeout
2021/05/07 11:54:04 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:54:04 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:39736->10.242.41.146:8300: i/o timeout
2021/05/07 11:54:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:05 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:54:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:07 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:54:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:09 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:54:09 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:54:09 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:54:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:15 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:39928->10.242.41.146:8300: i/o timeout
2021/05/07 11:54:16 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:54:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:17 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:39962->10.242.41.146:8300: i/o timeout
2021/05/07 11:54:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:21 [WARN] memberlist: Refuting a dead message (from: dev-consul-server-1)
2021/05/07 11:54:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:26 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:40102->10.242.41.146:8300: i/o timeout
2021/05/07 11:54:27 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:54:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:27 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:54:29 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:54:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:29 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:54:32 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:54:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:37 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:40306->10.242.41.146:8300: i/o timeout
2021/05/07 11:54:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:37 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:40322->10.242.41.146:8300: i/o timeout
2021/05/07 11:54:38 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:54:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:48 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:54:48 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:40492->10.242.41.146:8300: i/o timeout
2021/05/07 11:54:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:49 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:54:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:58 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:40678->10.242.41.146:8300: i/o timeout
2021/05/07 11:54:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:54:59 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:40694->10.242.41.146:8300: i/o timeout
2021/05/07 11:54:59 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:55:00 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:55:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:08 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:55:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:10 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:40892->10.242.41.146:8300: i/o timeout
2021/05/07 11:55:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:11 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:55:11 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:55:12 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:55:14 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:55:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:18 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:41032->10.242.41.146:8300: i/o timeout
2021/05/07 11:55:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:22 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:41102->10.242.41.146:8300: i/o timeout
2021/05/07 11:55:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:26 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:55:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:28 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:55:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:29 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:55:29 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.146:37822
2021/05/07 11:55:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:36 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:41352->10.242.41.146:8300: i/o timeout
2021/05/07 11:55:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:38 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:41394->10.242.41.146:8300: i/o timeout
2021/05/07 11:55:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:39 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:55:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:41 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:55:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:48 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:55:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:51 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:41624->10.242.41.146:8300: i/o timeout
2021/05/07 11:55:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:53 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:55:53 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:55:56 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:55:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:55:58 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:41752->10.242.41.146:8300: i/o timeout
2021/05/07 11:55:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:02 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:56:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:04 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 11:56:05 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:56:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:06 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500249179s
2021/05/07 11:56:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:09 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:56:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:09 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.93095554s
2021/05/07 11:56:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:11 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.431182036s
2021/05/07 11:56:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:15 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:41990->10.242.41.146:8300: i/o timeout
2021/05/07 11:56:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:15 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:56:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:19 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:42102->10.242.41.146:8300: i/o timeout
2021/05/07 11:56:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:19 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:56:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:25 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:42220->10.242.41.146:8300: i/o timeout
2021/05/07 11:56:26 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:56:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:29 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:56:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:35 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:56:35 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:56:36 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:42424->10.242.41.146:8300: i/o timeout
2021/05/07 11:56:37 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:56:38 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:56:38 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:56:38 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:56:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:39 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:42462->10.242.41.146:8300: i/o timeout
2021/05/07 11:56:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:47 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:42594->10.242.41.146:8300: i/o timeout
2021/05/07 11:56:47 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:56:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:49 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:56:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:40314->10.242.41.146:8302: i/o timeout
2021/05/07 11:56:49 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:56:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:57 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:42800->10.242.41.146:8300: i/o timeout
2021/05/07 11:56:58 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:56:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:56:59 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:42828->10.242.41.146:8300: i/o timeout
2021/05/07 11:56:59 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:56:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:40484->10.242.41.146:8302: i/o timeout
2021/05/07 11:56:59 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:57:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:08 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:42988->10.242.41.146:8300: i/o timeout
2021/05/07 11:57:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:09 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:57:09 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:57:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:40666->10.242.41.146:8302: i/o timeout
2021/05/07 11:57:09 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:57:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:15 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 11:57:15 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 11:57:15 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 11:57:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:17 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:57:17 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:57:19 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:43184->10.242.41.146:8300: i/o timeout
2021/05/07 11:57:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:40840->10.242.41.146:8302: i/o timeout
2021/05/07 11:57:19 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 11:57:19 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:43186->10.242.41.146:8300: i/o timeout
2021/05/07 11:57:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:57:20 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/07 11:57:20 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 11:57:21 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:57:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:30 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:57:31 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:43380->10.242.41.146:8300: i/o timeout
2021/05/07 11:57:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:32 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:57:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:34 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:57:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:39 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:57:40 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:43550->10.242.41.146:8300: i/o timeout
2021/05/07 11:57:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:42 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:43582->10.242.41.146:8300: i/o timeout
2021/05/07 11:57:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:44 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:57:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:50 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:57:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:54 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:57:54 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.137:41334
2021/05/07 11:57:54 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:43796->10.242.41.146:8300: i/o timeout
2021/05/07 11:57:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:57 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:57:59 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:57:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:57:59 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:58:00 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:43904->10.242.41.146:8300: i/o timeout
2021/05/07 11:58:02 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:58:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:07 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:44030->10.242.41.146:8300: i/o timeout
2021/05/07 11:58:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:10 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:58:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:12 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8302->10.242.41.146:54714: i/o timeout from=10.242.41.146:54714
2021/05/07 11:58:12 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8302->10.242.41.146:54714: i/o timeout from=10.242.41.146:54714
2021/05/07 11:58:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:13 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:58:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:19 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:58:20 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:44256->10.242.41.146:8300: i/o timeout
2021/05/07 11:58:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:23 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:44308->10.242.41.146:8300: i/o timeout
2021/05/07 11:58:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:30 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:58:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:34 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:58:34 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 11:58:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:40 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:44612->10.242.41.146:8300: i/o timeout
2021/05/07 11:58:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:41 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:58:41 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:58:44 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:58:44 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:44676->10.242.41.146:8300: i/o timeout
2021/05/07 11:58:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:51 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:58:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:55 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:58:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:58:59 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 11:59:01 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:44966->10.242.41.146:8300: i/o timeout
2021/05/07 11:59:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:05 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:45052->10.242.41.146:8300: i/o timeout
2021/05/07 11:59:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:11 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:59:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:16 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:59:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:21 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:45320->10.242.41.146:8300: i/o timeout
2021/05/07 11:59:23 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 11:59:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:23 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 11:59:26 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 11:59:26 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:45416->10.242.41.146:8300: i/o timeout
2021/05/07 11:59:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:31 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:59:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:37 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:59:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:39 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 11:59:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:41 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:45678->10.242.41.146:8300: i/o timeout
2021/05/07 11:59:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:44 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8301->10.242.41.146:56660: i/o timeout from=10.242.41.146:56660
2021/05/07 11:59:44 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8301->10.242.41.146:56660: i/o timeout from=10.242.41.146:56660
2021/05/07 11:59:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:47 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:45778->10.242.41.146:8300: i/o timeout
2021/05/07 11:59:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:51 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:59:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 11:59:58 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 11:59:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:01 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:46032->10.242.41.146:8300: i/o timeout
2021/05/07 12:00:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:04 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:00:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:05 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:00:05 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 12:00:08 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 12:00:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/07 12:00:08 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:46156->10.242.41.146:8300: i/o timeout
2021/05/07 12:00:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:12 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:00:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:19 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:00:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:22 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:46382->10.242.41.146:8300: i/o timeout
2021/05/07 12:00:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:29 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:46518->10.242.41.146:8300: i/o timeout
2021/05/07 12:00:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:32 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:00:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:36 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:00:36 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.146:38462
2021/05/07 12:00:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:40 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:00:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:42 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:46736->10.242.41.146:8300: i/o timeout
2021/05/07 12:00:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:47 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:00:47 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 12:00:49 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:00:50 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 12:00:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/05/07 12:00:50 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:46874->10.242.41.146:8300: i/o timeout
2021/05/07 12:00:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:52 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:00:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:00:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:01 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:01:02 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:47094->10.242.41.146:8300: i/o timeout
2021/05/07 12:01:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:11 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:47242->10.242.41.146:8300: i/o timeout
2021/05/07 12:01:12 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:01:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:22 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:01:22 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:47440->10.242.41.146:8300: i/o timeout
2021/05/07 12:01:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:29 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:01:29 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 12:01:29 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:01:32 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 12:01:32 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:47624->10.242.41.146:8300: i/o timeout
2021/05/07 12:01:33 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:01:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:34 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:01:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:43 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:47794->10.242.41.146:8300: i/o timeout
2021/05/07 12:01:43 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:01:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:53 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:47986->10.242.41.146:8300: i/o timeout
2021/05/07 12:01:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:53 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:01:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:01:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:03 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:48158->10.242.41.146:8300: i/o timeout
2021/05/07 12:02:04 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:02:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:11 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:02:11 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 12:02:13 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:02:14 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 12:02:14 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:48352->10.242.41.146:8300: i/o timeout
2021/05/07 12:02:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:23 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:48518->10.242.41.146:8300: i/o timeout
2021/05/07 12:02:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:25 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:02:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:33 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:02:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:35 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:48746->10.242.41.146:8300: i/o timeout
2021/05/07 12:02:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:43 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:48870->10.242.41.146:8300: i/o timeout
2021/05/07 12:02:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:46 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:02:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:50 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8301->10.242.41.146:57120: i/o timeout from=10.242.41.146:57120
2021/05/07 12:02:50 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8301->10.242.41.146:57120: i/o timeout from=10.242.41.146:57120
2021/05/07 12:02:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:53 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:02:53 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 12:02:53 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:02:56 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:49098->10.242.41.146:8300: i/o timeout
2021/05/07 12:02:56 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 12:02:56 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/05/07 12:02:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:02:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:03 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:49224->10.242.41.146:8300: i/o timeout
2021/05/07 12:03:04 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:03:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:06 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:03:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:09 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:03:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:14 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:03:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:16 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:49462->10.242.41.146:8300: i/o timeout
2021/05/07 12:03:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:24 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:49570->10.242.41.146:8300: i/o timeout
2021/05/07 12:03:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:28 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:03:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:33 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:03:33 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:03:34 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:03:34 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:03:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:35 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:03:35 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:03:35 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 12:03:35 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500307239s
2021/05/07 12:03:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:38 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.996144405s
2021/05/07 12:03:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:44652->10.242.41.146:8301: i/o timeout
2021/05/07 12:03:38 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:03:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:40 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.473604607s
2021/05/07 12:03:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:44684->10.242.41.146:8301: i/o timeout
2021/05/07 12:03:41 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:03:41 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:03:41 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:03:41 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 12:03:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:44 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:49842->10.242.41.146:8300: i/o timeout
2021/05/07 12:03:44 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:49940->10.242.41.146:8300: i/o timeout
2021/05/07 12:03:45 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:03:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:47712->10.242.41.146:8302: i/o timeout
2021/05/07 12:03:49 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:03:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:54 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:03:55 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:50150->10.242.41.146:8300: i/o timeout
2021/05/07 12:03:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:55 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:03:56 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:03:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:58 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500311417s
2021/05/07 12:03:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:03:59 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/05/07 12:04:01 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.942316569s
2021/05/07 12:04:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:03 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.403191929s
2021/05/07 12:04:04 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:50312->10.242.41.146:8300: i/o timeout
2021/05/07 12:04:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:06 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:50334->10.242.41.146:8300: i/o timeout
2021/05/07 12:04:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:07 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:04:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:48060->10.242.41.146:8302: i/o timeout
2021/05/07 12:04:09 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:04:10 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 12:04:10 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:04:10 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:04:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:13 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:04:13 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 12:04:14 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:04:16 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 12:04:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:17 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:50540->10.242.41.146:8300: i/o timeout
2021/05/07 12:04:18 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:04:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:24 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:50676->10.242.41.146:8300: i/o timeout
2021/05/07 12:04:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:25 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8302->10.242.41.146:55498: i/o timeout from=10.242.41.146:55498
2021/05/07 12:04:25 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8302->10.242.41.146:55498: i/o timeout from=10.242.41.146:55498
2021/05/07 12:04:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:28 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:50730->10.242.41.146:8300: i/o timeout
2021/05/07 12:04:28 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:04:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:33 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:04:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:35 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:04:35 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:04:35 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.41.146:38738
2021/05/07 12:04:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:38 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:50920->10.242.41.146:8300: i/o timeout
2021/05/07 12:04:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:39 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:04:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:45 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:51034->10.242.41.146:8300: i/o timeout
2021/05/07 12:04:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:49 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:51108->10.242.41.146:8300: i/o timeout
2021/05/07 12:04:50 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:04:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:55 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:04:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:04:55 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 12:04:55 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:04:58 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 12:04:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:00 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:51296->10.242.41.146:8300: i/o timeout
2021/05/07 12:05:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:01 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:05:01 [WARN] raft: Rejecting vote request from 10.242.41.146:8300 since we have a leader: 10.242.139.204:8300
2021/05/07 12:05:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:05 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:51390->10.242.41.146:8300: i/o timeout
2021/05/07 12:05:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:11 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:51488->10.242.41.146:8300: i/o timeout
2021/05/07 12:05:12 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:05:12 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/05/07 12:05:12 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:05:12 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:05:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:14 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:05:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:15 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:05:16 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.501549967s
2021/05/07 12:05:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:18 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.970842806s
2021/05/07 12:05:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:19 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:05:21 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.415488196s
2021/05/07 12:05:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:23 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8301->10.242.41.146:57462: i/o timeout from=10.242.41.146:57462
2021/05/07 12:05:23 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8301->10.242.41.146:57462: i/o timeout from=10.242.41.146:57462
2021/05/07 12:05:24 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:51680->10.242.41.146:8300: i/o timeout
2021/05/07 12:05:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:49296->10.242.41.146:8302: i/o timeout
2021/05/07 12:05:24 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:05:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:25 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:05:25 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:51740->10.242.41.146:8300: i/o timeout
2021/05/07 12:05:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:49486->10.242.41.146:8302: i/o timeout
2021/05/07 12:05:34 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:05:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:35 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:51934->10.242.41.146:8300: i/o timeout
2021/05/07 12:05:35 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:05:36 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:05:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:38 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:05:38 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:05:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:39 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:05:40 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:05:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:42 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 2.500212515s
2021/05/07 12:05:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:44 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 4.964095712s
2021/05/07 12:05:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:45 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:52100->10.242.41.146:8300: i/o timeout
2021/05/07 12:05:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:47 [WARN] raft: Failed to contact e90a9b84-7b02-fc1f-15b4-2419fe66fc38 in 7.453897216s
2021/05/07 12:05:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:49842->10.242.41.146:8302: i/o timeout
2021/05/07 12:05:49 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/05/07 12:05:50 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:52104->10.242.41.146:8300: i/o timeout
2021/05/07 12:05:50 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/05/07 12:05:50 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.41.146
2021/05/07 12:05:50 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/05/07 12:05:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:51 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:05:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:56 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:05:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:05:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:01 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:52384->10.242.41.146:8300: i/o timeout
2021/05/07 12:06:02 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:06:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:06 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:52466->10.242.41.146:8300: i/o timeout
2021/05/07 12:06:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:12 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:52566->10.242.41.146:8300: i/o timeout
2021/05/07 12:06:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:13 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:06:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:16 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:06:16 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 12:06:16 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:06:18 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 12:06:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:23 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:52756->10.242.41.146:8300: i/o timeout
2021/05/07 12:06:23 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:06:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:26 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:52820->10.242.41.146:8300: i/o timeout
2021/05/07 12:06:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:29 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:06:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:33 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:06:33 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:52960->10.242.41.146:8300: i/o timeout
2021/05/07 12:06:34 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:06:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:36 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:06:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:44 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:53156->10.242.41.146:8300: i/o timeout
2021/05/07 12:06:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:45 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:06:46 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:53180->10.242.41.146:8300: i/o timeout
2021/05/07 12:06:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:55 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:53348->10.242.41.146:8300: i/o timeout
2021/05/07 12:06:56 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:06:56 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:06:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:06:57 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:06:57 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 12:07:00 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 12:07:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:06 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:53532->10.242.41.146:8300: i/o timeout
2021/05/07 12:07:06 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:53536->10.242.41.146:8300: i/o timeout
2021/05/07 12:07:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:08 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:07:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:09 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-x5pmr 10.242.41.137:8301
2021/05/07 12:07:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:17 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:07:18 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:53734->10.242.41.146:8300: i/o timeout
2021/05/07 12:07:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:20 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:07:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:21 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.41.146
2021/05/07 12:07:21 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:07:21 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/05/07 12:07:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:48702->10.242.41.146:8301: i/o timeout
2021/05/07 12:07:26 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:07:26 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/05/07 12:07:26 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.41.146
2021/05/07 12:07:26 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.41.146:8300) (DC: dc1)
2021/05/07 12:07:26 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/05/07 12:07:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:27 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:53884->10.242.41.146:8300: i/o timeout
2021/05/07 12:07:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.139.204:48718->10.242.41.146:8301: i/o timeout
2021/05/07 12:07:29 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/05/07 12:07:30 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:53940->10.242.41.146:8300: i/o timeout
2021/05/07 12:07:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:33 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:07:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:37 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:07:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:39 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:07:39 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 12:07:42 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 12:07:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:43 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:54180->10.242.41.146:8300: i/o timeout
2021/05/07 12:07:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:47 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:54240->10.242.41.146:8300: i/o timeout
2021/05/07 12:07:48 [ERR] memberlist: failed to receive: read tcp 10.242.139.204:8302->10.242.41.146:55958: i/o timeout from=10.242.41.146:55958
2021/05/07 12:07:48 [ERR] memberlist: Failed to send error: write tcp 10.242.139.204:8302->10.242.41.146:55958: i/o timeout from=10.242.41.146:55958
2021/05/07 12:07:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:49 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:07:49 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.41.146:8301
2021/05/07 12:07:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:57 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:07:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:07:59 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:54460->10.242.41.146:8300: i/o timeout
2021/05/07 12:08:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:03 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.41.146:8302
2021/05/07 12:08:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:07 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:54608->10.242.41.146:8300: i/o timeout
2021/05/07 12:08:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:10 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:08:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:17 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:08:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:20 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:54822->10.242.41.146:8300: i/o timeout
2021/05/07 12:08:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:21 [ERR] yamux: keepalive failed: i/o deadline reached
2021/05/07 12:08:21 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/05/07 12:08:24 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/05/07 12:08:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:27 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:54962->10.242.41.146:8300: i/o timeout
2021/05/07 12:08:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:31 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:08:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:38 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:08:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:41 [ERR] raft: Failed to heartbeat to 10.242.41.146:8300: read tcp 10.242.139.204:55184->10.242.41.146:8300: i/o timeout
2021/05/07 12:08:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:48 [ERR] raft: Failed to AppendEntries to {Voter e90a9b84-7b02-fc1f-15b4-2419fe66fc38 10.242.41.146:8300}: read tcp 10.242.139.204:55314->10.242.41.146:8300: i/o timeout
2021/05/07 12:08:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:51 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38
2021/05/07 12:08:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/05/07 12:08:58 [WARN] Unable to get address for server id e90a9b84-7b02-fc1f-15b4-2419fe66fc38, using fallback address 10.242.41.146:8300: Could not find address for server id e90a9b84-