By type
+ CONSUL_BIND=
+ [ -n ]
+ CONSUL_CLIENT=
+ [ -n ]
+ CONSUL_DATA_DIR=/consul/data
+ CONSUL_CONFIG_DIR=/consul/config
+ [ -n ]
+ [ a = - ]
+ [ agent = agent ]
+ shift
+ set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
+ [ consul = consul ]
+ stat -c %u /consul/data
+ id -u consul
+ [ 100 != 100 ]
+ stat -c %u /consul/config
+ id -u consul
+ [ 100 != 100 ]
+ [ ! -z ]
+ exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
==> Starting Consul agent...
bootstrap_expect > 0: expecting 3 servers
==> Consul agent running!
Version: 'v1.0.6'
Node ID: 'fa409b74-3c2a-bdc6-3a5b-335beb268113'
Node name: 'dev-consul-server-2'
Datacenter: 'dc1' (Segment: '')
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.242.51.203 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2021/04/01 23:16:11 [INFO] raft: Initial configuration (index=0): []
2021/04/01 23:16:11 [INFO] raft: Node at 10.242.51.203:8300 [Follower] entering Follower state (Leader: "")
2021/04/01 23:16:11 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.51.203
2021/04/01 23:16:11 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.51.203
2021/04/01 23:16:11 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/01 23:16:11 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2021/04/01 23:16:11 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.51.203:8300) (DC: dc1)
2021/04/01 23:16:11 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2021/04/01 23:16:11 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
2021/04/01 23:16:11 [INFO] agent: started state syncer
2021/04/01 23:16:11 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer
2021/04/01 23:16:11 [INFO] agent: Joining LAN cluster...
2021/04/01 23:16:11 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/04/01 23:16:11 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.189.8
2021/04/01 23:16:11 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/01 23:16:11 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.189.8:8300) (DC: dc1)
2021/04/01 23:16:11 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-qzw6d 10.242.189.7
2021/04/01 23:16:11 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.242.51.203:8300,10.242.189.8:8300,10.242.241.18:8300
2021/04/01 23:16:11 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host
2021/04/01 23:16:11 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/01 23:16:11 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/01 23:16:11 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.189.8
2021/04/01 23:16:11 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/01 23:16:11 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/04/01 23:16:11 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/04/01 23:16:11 [INFO] agent: (LAN) joined: 1 Err:
2021/04/01 23:16:11 [INFO] agent: Join LAN completed. Synced with 1 initial agents
2021/04/01 23:16:15 [DEBUG] raft-net: 10.242.51.203:8300 accepted connection from: 10.242.189.8:48448
2021/04/01 23:16:15 [INFO] agent: Synced node info
2021/04/01 23:16:15 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/01 23:16:16 [DEBUG] raft-net: 10.242.51.203:8300 accepted connection from: 10.242.189.8:48456
==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=3f18c8c9-378b-c54f-8f70-ef9471bcb216&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2021/04/01 23:38:23 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:56484->10.242.241.18:8301: i/o timeout
2021/04/01 23:38:23 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/01 23:38:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:56522->10.242.241.18:8301: i/o timeout
2021/04/01 23:38:26 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/01 23:38:27 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/01 23:38:27 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/01 23:38:27 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/01 23:38:28 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/01 23:38:28 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/01 23:38:29 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/01 23:39:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:57240->10.242.241.18:8301: i/o timeout
2021/04/01 23:39:20 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/01 23:55:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:41060->10.242.241.18:8301: i/o timeout
2021/04/01 23:55:04 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/01 23:55:05 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/01 23:55:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:41156->10.242.241.18:8301: i/o timeout
2021/04/01 23:55:11 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/01 23:55:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:41172->10.242.241.18:8301: i/o timeout
2021/04/01 23:55:14 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/01 23:55:14 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/01 23:55:14 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/01 23:55:14 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/01 23:55:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:34464->10.242.241.18:8302: i/o timeout
2021/04/01 23:55:16 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/01 23:55:16 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/01 23:55:16 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/01 23:55:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:41260->10.242.241.18:8301: i/o timeout
2021/04/01 23:55:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:41454->10.242.241.18:8301: i/o timeout
2021/04/01 23:55:34 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/01 23:55:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:41486->10.242.241.18:8301: i/o timeout
2021/04/01 23:55:37 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/01 23:55:37 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/01 23:55:37 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/01 23:55:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:34862->10.242.241.18:8302: i/o timeout
2021/04/01 23:55:46 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/01 23:55:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:34922->10.242.241.18:8302: i/o timeout
2021/04/01 23:55:56 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/01 23:56:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:35178->10.242.241.18:8302: i/o timeout
2021/04/01 23:56:11 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/01 23:56:12 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/01 23:56:12 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/01 23:56:12 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/01 23:56:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:35234->10.242.241.18:8302: i/o timeout
2021/04/01 23:56:21 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/01 23:56:37 [ERR] memberlist: failed to receive: read tcp 10.242.51.203:8302->10.242.241.18:36708: i/o timeout from=10.242.241.18:36708
2021/04/01 23:56:37 [ERR] memberlist: Failed to send error: write tcp 10.242.51.203:8302->10.242.241.18:36708: i/o timeout from=10.242.241.18:36708
2021/04/01 23:56:38 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/01 23:56:38 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/01 23:56:38 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/01 23:56:41 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/01 23:56:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:42340->10.242.241.18:8301: i/o timeout
2021/04/01 23:56:43 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/01 23:56:45 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/01 23:56:45 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/01 23:56:45 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/01 23:57:20 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/01 23:57:40 [ERR] memberlist: failed to receive: read tcp 10.242.51.203:8302->10.242.241.18:37348: i/o timeout from=10.242.241.18:37348
2021/04/01 23:57:40 [ERR] memberlist: Failed to send error: write tcp 10.242.51.203:8302->10.242.241.18:37348: i/o timeout from=10.242.241.18:37348
2021/04/01 23:57:48 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/01 23:57:49 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/01 23:58:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:36612->10.242.241.18:8302: i/o timeout
2021/04/01 23:58:06 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/01 23:58:11 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/01 23:58:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:36740->10.242.241.18:8302: i/o timeout
2021/04/01 23:58:16 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/01 23:58:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:36878->10.242.241.18:8302: i/o timeout
2021/04/01 23:58:26 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/01 23:58:27 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/01 23:58:27 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/01 23:58:27 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/01 23:59:11 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/01 23:59:21 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:00:16 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:00:16 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:00:19 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/04/02 00:00:19 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:00:19 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:00:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:45470->10.242.241.18:8301: i/o timeout
2021/04/02 00:00:50 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/02 00:02:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:46588->10.242.241.18:8301: i/o timeout
2021/04/02 00:02:19 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:02:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:46720->10.242.241.18:8301: i/o timeout
2021/04/02 00:02:29 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:02:31 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:02:31 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:02:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:46760->10.242.241.18:8301: i/o timeout
2021/04/02 00:02:32 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:02:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:40046->10.242.241.18:8302: i/o timeout
2021/04/02 00:02:36 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:02:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:40110->10.242.241.18:8302: i/o timeout
2021/04/02 00:02:46 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:03:01 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:03:06 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/02 00:03:06 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:03:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:40466->10.242.241.18:8302: i/o timeout
2021/04/02 00:03:06 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:03:06 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:03:06 [ERR] memberlist: failed to receive: read tcp 10.242.51.203:8302->10.242.241.18:42120: i/o timeout from=10.242.241.18:42120
2021/04/02 00:03:06 [ERR] memberlist: Failed to send error: write tcp 10.242.51.203:8302->10.242.241.18:42120: i/o timeout from=10.242.241.18:42120
2021/04/02 00:03:41 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:04:11 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:05:11 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:06:11 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:07:11 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:08:11 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:08:51 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:09:11 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:10:11 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:10:43 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:10:43 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:10:43 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:10:43 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:10:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:53622->10.242.241.18:8301: i/o timeout
2021/04/02 00:10:47 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:10:51 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:10:51 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:10:51 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:10:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:53698->10.242.241.18:8301: i/o timeout
2021/04/02 00:10:52 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:11:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:47064->10.242.241.18:8302: i/o timeout
2021/04/02 00:11:01 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:11:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:47220->10.242.241.18:8302: i/o timeout
2021/04/02 00:11:11 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:11:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:47356->10.242.241.18:8302: i/o timeout
2021/04/02 00:11:21 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:11:27 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/02 00:11:27 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:11:27 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:11:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:47508->10.242.241.18:8302: i/o timeout
2021/04/02 00:11:31 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:11:38 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:11:38 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:11:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:54450->10.242.241.18:8301: i/o timeout
2021/04/02 00:11:44 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:11:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:54466->10.242.241.18:8301: i/o timeout
2021/04/02 00:11:47 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:11:47 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:11:47 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:11:47 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:12:09 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/04/02 00:12:20 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:12:20 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:12:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:48298->10.242.241.18:8302: i/o timeout
2021/04/02 00:12:26 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:12:30 [ERR] memberlist: failed to receive: read tcp 10.242.51.203:8301->10.242.241.18:33760: i/o timeout from=10.242.241.18:33760
2021/04/02 00:12:30 [ERR] memberlist: Failed to send error: write tcp 10.242.51.203:8301->10.242.241.18:33760: i/o timeout from=10.242.241.18:33760
2021/04/02 00:12:31 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:12:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:48498->10.242.241.18:8302: i/o timeout
2021/04/02 00:12:41 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:12:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:48644->10.242.241.18:8302: i/o timeout
2021/04/02 00:12:51 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:12:54 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:12:54 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:12:56 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/02 00:12:56 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:12:56 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:12:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:55476->10.242.241.18:8301: i/o timeout
2021/04/02 00:12:56 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:12:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:55518->10.242.241.18:8301: i/o timeout
2021/04/02 00:12:59 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:13:00 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:13:00 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:13:00 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:13:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:48790->10.242.241.18:8302: i/o timeout
2021/04/02 00:13:01 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:13:41 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:14:01 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:14:11 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:15:02 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:15:02 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.241.18:60558
2021/04/02 00:15:11 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:15:27 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:15:27 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:15:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:57766->10.242.241.18:8301: i/o timeout
2021/04/02 00:15:29 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:15:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:57796->10.242.241.18:8301: i/o timeout
2021/04/02 00:15:32 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:15:33 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:15:33 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:15:37 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:15:37 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:15:44 [ERR] memberlist: failed to receive: read tcp 10.242.51.203:8302->10.242.241.18:47604: i/o timeout from=10.242.241.18:47604
2021/04/02 00:15:44 [ERR] memberlist: Failed to send error: write tcp 10.242.51.203:8302->10.242.241.18:47604: i/o timeout from=10.242.241.18:47604
2021/04/02 00:15:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:51256->10.242.241.18:8302: i/o timeout
2021/04/02 00:15:46 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:15:47 [ERR] memberlist: failed to receive: read tcp 10.242.51.203:8301->10.242.241.18:35598: i/o timeout from=10.242.241.18:35598
2021/04/02 00:15:47 [ERR] memberlist: Failed to send error: write tcp 10.242.51.203:8301->10.242.241.18:35598: i/o timeout from=10.242.241.18:35598
2021/04/02 00:15:48 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/02 00:16:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:51456->10.242.241.18:8302: i/o timeout
2021/04/02 00:16:01 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:16:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:51604->10.242.241.18:8302: i/o timeout
2021/04/02 00:16:11 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:16:16 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/02 00:16:16 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:16:16 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:16:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:51742->10.242.241.18:8302: i/o timeout
2021/04/02 00:16:21 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:16:37 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:16:46 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:16:46 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:16:49 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.189.8
2021/04/02 00:16:49 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/04/02 00:16:49 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.189.8
2021/04/02 00:16:49 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/04/02 00:16:49 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:16:49 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:16:55 [DEBUG] raft-net: 10.242.51.203:8300 accepted connection from: 10.242.241.18:34074
2021/04/02 00:16:55 [WARN] raft: Rejecting vote request from 10.242.241.18:8300 since we have a leader: 10.242.189.8:8300
2021/04/02 00:17:03 [WARN] raft: Heartbeat timeout from "10.242.189.8:8300" reached, starting election
2021/04/02 00:17:03 [INFO] raft: Node at 10.242.51.203:8300 [Candidate] entering Candidate state in term 3
2021/04/02 00:17:03 [INFO] raft: Election won. Tally: 2
2021/04/02 00:17:03 [INFO] raft: Node at 10.242.51.203:8300 [Leader] entering Leader state
2021/04/02 00:17:03 [INFO] consul: cluster leadership acquired
2021/04/02 00:17:03 [INFO] raft: Added peer 89c1a268-ee85-dfa0-749a-f6802dc15803, starting replication
2021/04/02 00:17:03 [INFO] consul: New leader elected: dev-consul-server-2
2021/04/02 00:17:03 [INFO] raft: Added peer ab3e202d-9a1e-5268-25eb-4c4881c4e864, starting replication
2021/04/02 00:17:03 [INFO] raft: pipelining replication to peer {Voter 89c1a268-ee85-dfa0-749a-f6802dc15803 10.242.189.8:8300}
2021/04/02 00:17:03 [WARN] raft: Rejecting vote request from 10.242.241.18:8300 since we have a leader: 10.242.51.203:8300
2021/04/02 00:17:03 [ERR] raft: peer {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300} has newer term, stopping replication
2021/04/02 00:17:03 [INFO] raft: Node at 10.242.51.203:8300 [Follower] entering Follower state (Leader: "")
2021/04/02 00:17:03 [INFO] raft: aborting pipeline replication to peer {Voter 89c1a268-ee85-dfa0-749a-f6802dc15803 10.242.189.8:8300}
2021/04/02 00:17:03 [INFO] consul: cluster leadership lost
2021/04/02 00:17:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:59166->10.242.241.18:8301: i/o timeout
2021/04/02 00:17:06 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:17:06 [WARN] consul.coordinate: Batch update failed: node is not the leader
2021/04/02 00:17:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:59226->10.242.241.18:8301: i/o timeout
2021/04/02 00:17:10 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:17:10 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:17:10 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:17:10 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:17:10 [WARN] raft: Rejecting vote request from 10.242.241.18:8300 since our last term is greater (3, 2)
2021/04/02 00:17:10 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:17:10 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:17:11 [ERR] agent: failed to sync remote state: No cluster leader
2021/04/02 00:17:13 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/04/02 00:17:13 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/02 00:17:13 [INFO] raft: Node at 10.242.51.203:8300 [Candidate] entering Candidate state in term 6
2021/04/02 00:17:13 [INFO] raft: Election won. Tally: 2
2021/04/02 00:17:13 [INFO] raft: Node at 10.242.51.203:8300 [Leader] entering Leader state
2021/04/02 00:17:13 [INFO] raft: Added peer 89c1a268-ee85-dfa0-749a-f6802dc15803, starting replication
2021/04/02 00:17:13 [INFO] raft: Added peer ab3e202d-9a1e-5268-25eb-4c4881c4e864, starting replication
2021/04/02 00:17:13 [INFO] consul: cluster leadership acquired
2021/04/02 00:17:13 [INFO] consul: New leader elected: dev-consul-server-2
2021/04/02 00:17:13 [INFO] raft: pipelining replication to peer {Voter 89c1a268-ee85-dfa0-749a-f6802dc15803 10.242.189.8:8300}
2021/04/02 00:17:15 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.501576554s
2021/04/02 00:17:16 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:17:17 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/04/02 00:17:17 [WARN] raft: AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300} rejected, sending older logs (next: 681)
2021/04/02 00:17:17 [INFO] raft: pipelining replication to peer {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}
2021/04/02 00:17:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:17:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:17:21 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500217296s
2021/04/02 00:17:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:59416->10.242.241.18:8301: i/o timeout
2021/04/02 00:17:22 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:17:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:17:23 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/02 00:17:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:59502->10.242.241.18:8301: i/o timeout
2021/04/02 00:17:28 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:17:28 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:17:28 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/02 00:17:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:17:30 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.50019722s
2021/04/02 00:17:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:59534->10.242.241.18:8301: i/o timeout
2021/04/02 00:17:31 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:17:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:17:32 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:17:32 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:17:32 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:17:32 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/02 00:17:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:17:33 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 4.992558619s
2021/04/02 00:17:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:17:35 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:17:35 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:17:35 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/02 00:17:38 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:17:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:17:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:59680->10.242.241.18:8301: i/o timeout
2021/04/02 00:17:40 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:17:40 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:17:40 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:17:40 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/02 00:17:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:52932->10.242.241.18:8302: i/o timeout
2021/04/02 00:17:41 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:17:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:17:41 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:17:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:59694->10.242.241.18:8301: i/o timeout
2021/04/02 00:17:43 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:17:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:17:43 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500266761s
2021/04/02 00:17:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:17:45 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 4.916686313s
2021/04/02 00:17:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:17:47 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:17:47 [INFO] raft: aborting pipeline replication to peer {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}
2021/04/02 00:17:47 [ERR] memberlist: failed to receive: read tcp 10.242.51.203:8301->10.242.241.18:37328: i/o timeout from=10.242.241.18:37328
2021/04/02 00:17:47 [ERR] memberlist: Failed to send error: write tcp 10.242.51.203:8301->10.242.241.18:37328: i/o timeout from=10.242.241.18:37328
2021/04/02 00:17:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:17:50 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500195943s
2021/04/02 00:17:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:53084->10.242.241.18:8302: i/o timeout
2021/04/02 00:17:51 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:17:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:17:51 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:34848->10.242.241.18:8300: i/o timeout
2021/04/02 00:17:52 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:17:52 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 4.986997112s
2021/04/02 00:17:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:17:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:17:55 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 7.439092617s
2021/04/02 00:17:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:17:57 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:17:57 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.241.18:34426
2021/04/02 00:17:57 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:34686->10.242.241.18:8300: i/o timeout
2021/04/02 00:17:58 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:17:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:53232->10.242.241.18:8302: i/o timeout
2021/04/02 00:18:01 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:18:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:02 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:35430->10.242.241.18:8300: i/o timeout
2021/04/02 00:18:02 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:18:02 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/02 00:18:02 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:18:02 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/02 00:18:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:60058->10.242.241.18:8301: i/o timeout
2021/04/02 00:18:05 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:18:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:08 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:35518->10.242.241.18:8300: i/o timeout
2021/04/02 00:18:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:60118->10.242.241.18:8301: i/o timeout
2021/04/02 00:18:09 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:18:09 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:18:09 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:18:09 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:18:09 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/02 00:18:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:09 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500178854s
2021/04/02 00:18:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:12 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 4.946593601s
2021/04/02 00:18:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:14 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 7.414835006s
2021/04/02 00:18:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:18 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:35604->10.242.241.18:8300: i/o timeout
2021/04/02 00:18:18 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:18:18 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:35678->10.242.241.18:8300: i/o timeout
2021/04/02 00:18:18 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:18:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:53532->10.242.241.18:8302: i/o timeout
2021/04/02 00:18:21 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:18:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:28 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:18:28 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:18:28 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:35816->10.242.241.18:8300: i/o timeout
2021/04/02 00:18:28 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:18:28 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:35832->10.242.241.18:8300: i/o timeout
2021/04/02 00:18:29 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:18:30 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:18:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:53624->10.242.241.18:8302: i/o timeout
2021/04/02 00:18:31 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:18:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:37 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:18:37 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:18:37 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/02 00:18:38 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:35988->10.242.241.18:8300: i/o timeout
2021/04/02 00:18:38 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/04/02 00:18:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:39 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:36014->10.242.241.18:8300: i/o timeout
2021/04/02 00:18:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:60590->10.242.241.18:8301: i/o timeout
2021/04/02 00:18:40 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:18:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:53842->10.242.241.18:8302: i/o timeout
2021/04/02 00:18:41 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:18:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:60644->10.242.241.18:8301: i/o timeout
2021/04/02 00:18:43 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:18:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:43 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:18:43 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:18:43 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/02 00:18:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:48 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:36126->10.242.241.18:8300: i/o timeout
2021/04/02 00:18:48 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:18:48 [ERR] memberlist: failed to receive: read tcp 10.242.51.203:8301->10.242.241.18:38094: i/o timeout from=10.242.241.18:38094
2021/04/02 00:18:48 [ERR] memberlist: Failed to send error: write tcp 10.242.51.203:8301->10.242.241.18:38094: i/o timeout from=10.242.241.18:38094
2021/04/02 00:18:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:50 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:36178->10.242.241.18:8300: i/o timeout
2021/04/02 00:18:51 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/02 00:18:51 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:18:51 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:18:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:51 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:18:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:53994->10.242.241.18:8302: i/o timeout
2021/04/02 00:18:56 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:18:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:18:58 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:36288->10.242.241.18:8300: i/o timeout
2021/04/02 00:18:58 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:18:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:01 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:36340->10.242.241.18:8300: i/o timeout
2021/04/02 00:19:02 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:19:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:08 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:36444->10.242.241.18:8300: i/o timeout
2021/04/02 00:19:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:09 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:19:09 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:19:09 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:19:12 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:19:12 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:36500->10.242.241.18:8300: i/o timeout
2021/04/02 00:19:13 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:19:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:19 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:36614->10.242.241.18:8300: i/o timeout
2021/04/02 00:19:20 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:19:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:23 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:36664->10.242.241.18:8300: i/o timeout
2021/04/02 00:19:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:24 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:19:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:30 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:36782->10.242.241.18:8300: i/o timeout
2021/04/02 00:19:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:33 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:19:34 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:36842->10.242.241.18:8300: i/o timeout
2021/04/02 00:19:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:35 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:19:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:43 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:36962->10.242.241.18:8300: i/o timeout
2021/04/02 00:19:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:45 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:19:46 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500207756s
2021/04/02 00:19:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:48 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:19:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:49 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 4.951937068s
2021/04/02 00:19:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:51 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:19:51 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 7.444685002s
2021/04/02 00:19:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:55 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:37002->10.242.241.18:8300: i/o timeout
2021/04/02 00:19:56 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:19:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:19:58 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:37190->10.242.241.18:8300: i/o timeout
2021/04/02 00:19:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:06 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:37300->10.242.241.18:8300: i/o timeout
2021/04/02 00:20:06 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:20:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:08 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:20:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:16 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:37454->10.242.241.18:8300: i/o timeout
2021/04/02 00:20:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:17 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:20:18 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:37488->10.242.241.18:8300: i/o timeout
2021/04/02 00:20:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:24 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:20:24 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:20:26 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:20:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:27 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:37616->10.242.241.18:8300: i/o timeout
2021/04/02 00:20:28 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:20:28 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:20:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:38 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:37790->10.242.241.18:8300: i/o timeout
2021/04/02 00:20:38 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:37792->10.242.241.18:8300: i/o timeout
2021/04/02 00:20:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:39 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:20:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:46 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:20:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:49 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:20:49 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:37944->10.242.241.18:8300: i/o timeout
2021/04/02 00:20:50 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:20:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:20:59 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:38082->10.242.241.18:8300: i/o timeout
2021/04/02 00:21:00 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:38102->10.242.241.18:8300: i/o timeout
2021/04/02 00:21:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:01 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:21:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:05 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:21:05 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:21:08 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:21:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:09 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:21:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:11 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:38270->10.242.241.18:8300: i/o timeout
2021/04/02 00:21:12 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:21:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:19 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:38398->10.242.241.18:8300: i/o timeout
2021/04/02 00:21:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:22 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:38442->10.242.241.18:8300: i/o timeout
2021/04/02 00:21:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:24 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:21:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:29 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:21:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:34 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:38630->10.242.241.18:8300: i/o timeout
2021/04/02 00:21:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:37 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:21:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:39 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:38702->10.242.241.18:8300: i/o timeout
2021/04/02 00:21:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:47 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:21:47 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:21:47 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:38818->10.242.241.18:8300: i/o timeout
2021/04/02 00:21:49 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:21:50 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:21:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:53 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:21:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:21:59 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:39004->10.242.241.18:8300: i/o timeout
2021/04/02 00:22:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:01 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:22:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:03 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:39066->10.242.241.18:8300: i/o timeout
2021/04/02 00:22:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:10 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:22:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:15 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:22:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:20 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:39322->10.242.241.18:8300: i/o timeout
2021/04/02 00:22:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:25 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:39394->10.242.241.18:8300: i/o timeout
2021/04/02 00:22:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:29 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:22:29 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:22:30 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:22:32 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:22:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:36 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:22:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:40 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:39632->10.242.241.18:8300: i/o timeout
2021/04/02 00:22:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:46 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:39716->10.242.241.18:8300: i/o timeout
2021/04/02 00:22:46 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:22:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:50 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:22:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:56 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:22:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:22:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:00 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:39948->10.242.241.18:8300: i/o timeout
2021/04/02 00:23:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:06 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:40026->10.242.241.18:8300: i/o timeout
2021/04/02 00:23:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:10 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:23:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:11 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:23:11 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:23:11 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:23:14 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:23:14 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/02 00:23:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:18 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:23:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:20 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:40252->10.242.241.18:8300: i/o timeout
2021/04/02 00:23:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:28 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:40356->10.242.241.18:8300: i/o timeout
2021/04/02 00:23:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:31 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:23:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:38 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:23:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:41 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:40554->10.242.241.18:8300: i/o timeout
2021/04/02 00:23:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:46 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:23:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:48 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:40666->10.242.241.18:8300: i/o timeout
2021/04/02 00:23:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:51 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:23:51 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:23:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:53 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:23:53 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:23:56 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:23:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:23:59 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:24:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:01 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:40866->10.242.241.18:8300: i/o timeout
2021/04/02 00:24:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:09 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:40984->10.242.241.18:8300: i/o timeout
2021/04/02 00:24:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:11 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:24:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:20 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:24:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:21 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:41172->10.242.241.18:8300: i/o timeout
2021/04/02 00:24:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:30 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:41310->10.242.241.18:8300: i/o timeout
2021/04/02 00:24:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:31 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:24:31 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:24:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:35 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:24:35 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:24:38 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:24:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:41 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:24:41 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:41488->10.242.241.18:8300: i/o timeout
2021/04/02 00:24:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:51 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:41634->10.242.241.18:8300: i/o timeout
2021/04/02 00:24:52 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:24:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:24:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:02 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:41782->10.242.241.18:8300: i/o timeout
2021/04/02 00:25:02 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:25:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:12 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:25:12 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:41944->10.242.241.18:8300: i/o timeout
2021/04/02 00:25:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:17 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:25:17 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:25:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:25:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:22 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:42074->10.242.241.18:8300: i/o timeout
2021/04/02 00:25:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:23 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:25:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:32 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:25:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:33 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:42252->10.242.241.18:8300: i/o timeout
2021/04/02 00:25:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:42 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:42372->10.242.241.18:8300: i/o timeout
2021/04/02 00:25:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:44 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:25:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:52 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:25:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:54 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:42554->10.242.241.18:8300: i/o timeout
2021/04/02 00:25:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:25:59 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:25:59 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:26:02 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:26:02 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/02 00:26:02 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:42668->10.242.241.18:8300: i/o timeout
2021/04/02 00:26:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:05 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:26:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:11 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:26:13 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:26:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:15 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:42862->10.242.241.18:8300: i/o timeout
2021/04/02 00:26:16 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:26:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:23 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:42962->10.242.241.18:8300: i/o timeout
2021/04/02 00:26:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:26 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:26:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:33 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:26:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:36 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:43164->10.242.241.18:8300: i/o timeout
2021/04/02 00:26:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:41 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:26:41 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:26:43 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:43258->10.242.241.18:8300: i/o timeout
2021/04/02 00:26:44 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:26:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:47 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:26:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:53 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:26:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:26:57 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:43466->10.242.241.18:8300: i/o timeout
2021/04/02 00:26:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:03 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:43546->10.242.241.18:8300: i/o timeout
2021/04/02 00:27:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:08 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:27:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:13 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:27:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:18 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:43768->10.242.241.18:8300: i/o timeout
2021/04/02 00:27:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:23 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:27:23 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:27:23 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:43844->10.242.241.18:8300: i/o timeout
2021/04/02 00:27:26 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:27:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:29 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:27:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:34 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:27:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:39 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:44074->10.242.241.18:8300: i/o timeout
2021/04/02 00:27:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:44 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:44140->10.242.241.18:8300: i/o timeout
2021/04/02 00:27:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:46 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:27:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:50 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:27:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:51 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:27:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:54 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:27:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:27:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:00 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:44376->10.242.241.18:8300: i/o timeout
2021/04/02 00:28:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:04 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:44436->10.242.241.18:8300: i/o timeout
2021/04/02 00:28:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:05 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:28:05 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:28:08 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:28:08 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/02 00:28:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:11 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:28:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:14 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:28:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:21 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:44686->10.242.241.18:8300: i/o timeout
2021/04/02 00:28:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:24 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:44728->10.242.241.18:8300: i/o timeout
2021/04/02 00:28:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:31 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:28:32 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:28:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:34 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:28:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:42 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:44994->10.242.241.18:8300: i/o timeout
2021/04/02 00:28:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:44 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:45024->10.242.241.18:8300: i/o timeout
2021/04/02 00:28:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:47 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:28:47 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:28:50 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:28:50 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/02 00:28:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:53 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:28:54 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:28:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:28:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:03 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:45296->10.242.241.18:8300: i/o timeout
2021/04/02 00:29:04 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:45322->10.242.241.18:8300: i/o timeout
2021/04/02 00:29:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:14 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:29:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:15 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:29:16 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:29:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:24 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:45610->10.242.241.18:8300: i/o timeout
2021/04/02 00:29:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:25 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:45624->10.242.241.18:8300: i/o timeout
2021/04/02 00:29:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:29 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:29:29 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:29:32 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:29:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:35 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:29:35 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:29:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:41 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:29:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:45 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:45920->10.242.241.18:8300: i/o timeout
2021/04/02 00:29:45 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:45924->10.242.241.18:8300: i/o timeout
2021/04/02 00:29:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:55 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:29:56 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:29:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:29:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:30:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:30:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:30:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:30:05 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:46212->10.242.241.18:8300: i/o timeout
2021/04/02 00:30:06 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:46220->10.242.241.18:8300: i/o timeout
2021/04/02 00:30:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:30:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:30:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:30:11 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:30:11 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:30:14 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:30:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:30:15 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:30:16 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:30:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:30:17 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:30:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:30:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:30:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:30:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:30:25 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:46510->10.242.241.18:8300: i/o timeout
2021/04/02 00:30:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:30:27 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:46534->10.242.241.18:8300: i/o timeout
2021/04/02 00:30:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:30:29 [WARN] raft: Rejecting vote request from 10.242.241.18:8300 since we have a leader: 10.242.51.203:8300
2021/04/02 00:30:29 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:30:29 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:30:29 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/02 00:30:30 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:30:30 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:30:36 [INFO] raft: pipelining replication to peer {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}
2021/04/02 00:33:58 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:33:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:46066->10.242.241.18:8301: i/o timeout
2021/04/02 00:33:59 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:34:08 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:34:10 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:34:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:46258->10.242.241.18:8301: i/o timeout
2021/04/02 00:34:11 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:34:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:34:14 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:34:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:46320->10.242.241.18:8301: i/o timeout
2021/04/02 00:34:15 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:34:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:34:16 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.50436777s
2021/04/02 00:34:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:34:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:46360->10.242.241.18:8301: i/o timeout
2021/04/02 00:34:18 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:34:18 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:34:18 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:34:18 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/02 00:34:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:34:20 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500239521s
2021/04/02 00:34:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:39652->10.242.241.18:8302: i/o timeout
2021/04/02 00:34:21 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:34:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:34:21 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:34:22 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 4.98156261s
2021/04/02 00:34:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:34:23 [ERR] memberlist: Push/Pull with dev-consul-server-1 failed: read tcp 10.242.51.203:46294->10.242.241.18:8301: i/o timeout
2021/04/02 00:34:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:34:25 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 7.47857577s
2021/04/02 00:34:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:34:27 [INFO] raft: aborting pipeline replication to peer {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}
2021/04/02 00:34:27 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:34:28 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:46844->10.242.241.18:8300: i/o timeout
2021/04/02 00:34:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:34:29 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:34:30 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.504228673s
2021/04/02 00:34:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:39814->10.242.241.18:8302: i/o timeout
2021/04/02 00:34:31 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:34:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:34:32 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 4.946526422s
2021/04/02 00:34:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:34:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:34:35 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 7.35667726s
2021/04/02 00:34:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:34:37 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:50322->10.242.241.18:8300: i/o timeout
2021/04/02 00:34:37 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:34:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:34:39 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:50350->10.242.241.18:8300: i/o timeout
2021/04/02 00:34:40 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:34:40 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:34:40 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.241.18:38208
2021/04/02 00:34:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:34:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:34:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:34:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:40056->10.242.241.18:8302: i/o timeout
2021/04/02 00:34:46 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:34:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:34:47 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:50478->10.242.241.18:8300: i/o timeout
2021/04/02 00:34:47 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:34:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:34:50 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:50520->10.242.241.18:8300: i/o timeout
2021/04/02 00:34:50 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:34:51 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/02 00:34:51 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:34:51 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:34:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:34:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:34:53 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:34:53 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:34:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:40206->10.242.241.18:8302: i/o timeout
2021/04/02 00:34:56 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:34:56 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:34:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:34:57 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:50640->10.242.241.18:8300: i/o timeout
2021/04/02 00:34:57 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:34:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:00 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:35:00 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:50690->10.242.241.18:8300: i/o timeout
2021/04/02 00:35:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:01 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:35:01 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:35:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:07 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:50792->10.242.241.18:8300: i/o timeout
2021/04/02 00:35:08 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:35:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:11 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:50858->10.242.241.18:8300: i/o timeout
2021/04/02 00:35:12 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:35:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:18 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:50952->10.242.241.18:8300: i/o timeout
2021/04/02 00:35:18 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:35:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:22 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:51008->10.242.241.18:8300: i/o timeout
2021/04/02 00:35:22 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:35:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:28 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:51100->10.242.241.18:8300: i/o timeout
2021/04/02 00:35:28 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:35:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:32 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:51176->10.242.241.18:8300: i/o timeout
2021/04/02 00:35:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:34 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:35:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:35 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:35:35 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:35:38 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:35:38 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/02 00:35:38 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:51264->10.242.241.18:8300: i/o timeout
2021/04/02 00:35:38 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:35:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:41 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:35:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:44 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:51354->10.242.241.18:8300: i/o timeout
2021/04/02 00:35:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:45 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:35:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:48 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:51426->10.242.241.18:8300: i/o timeout
2021/04/02 00:35:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:49 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:35:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:55 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:51574->10.242.241.18:8300: i/o timeout
2021/04/02 00:35:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:57 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:35:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:35:59 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:51632->10.242.241.18:8300: i/o timeout
2021/04/02 00:36:00 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:36:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:07 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:51764->10.242.241.18:8300: i/o timeout
2021/04/02 00:36:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:10 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:51828->10.242.241.18:8300: i/o timeout
2021/04/02 00:36:10 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:36:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:13 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:36:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:17 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:36:17 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:36:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:36:20 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:52014->10.242.241.18:8300: i/o timeout
2021/04/02 00:36:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:23 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:52038->10.242.241.18:8300: i/o timeout
2021/04/02 00:36:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:26 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:36:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:28 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:36:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:30 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:36:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:36 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:52274->10.242.241.18:8300: i/o timeout
2021/04/02 00:36:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:38 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:52304->10.242.241.18:8300: i/o timeout
2021/04/02 00:36:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:47 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:36:48 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:36:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:51 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:36:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:57 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:52588->10.242.241.18:8300: i/o timeout
2021/04/02 00:36:58 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:52604->10.242.241.18:8300: i/o timeout
2021/04/02 00:36:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:36:59 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:36:59 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:37:02 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:37:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:08 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:37:08 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:37:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:18 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:52908->10.242.241.18:8300: i/o timeout
2021/04/02 00:37:18 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:52912->10.242.241.18:8300: i/o timeout
2021/04/02 00:37:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:28 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:37:29 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:37:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:31 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:37:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:38 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:53218->10.242.241.18:8300: i/o timeout
2021/04/02 00:37:39 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:53220->10.242.241.18:8300: i/o timeout
2021/04/02 00:37:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:41 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:37:41 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:37:44 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:37:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:49 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:37:50 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:37:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:37:59 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:53508->10.242.241.18:8300: i/o timeout
2021/04/02 00:38:00 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:53532->10.242.241.18:8300: i/o timeout
2021/04/02 00:38:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:09 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:38:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:11 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:38:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:19 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:53816->10.242.241.18:8300: i/o timeout
2021/04/02 00:38:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:21 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:53852->10.242.241.18:8300: i/o timeout
2021/04/02 00:38:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:23 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:38:23 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:38:26 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:38:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:29 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:38:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:32 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:38:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:39 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:54112->10.242.241.18:8300: i/o timeout
2021/04/02 00:38:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:42 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:54152->10.242.241.18:8300: i/o timeout
2021/04/02 00:38:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:49 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:38:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:53 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:38:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:38:59 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:54406->10.242.241.18:8300: i/o timeout
2021/04/02 00:39:00 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:39:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:03 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:54454->10.242.241.18:8300: i/o timeout
2021/04/02 00:39:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:05 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:39:05 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:39:08 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:39:08 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/02 00:39:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:10 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:39:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:14 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:39:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:20 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:54714->10.242.241.18:8300: i/o timeout
2021/04/02 00:39:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:24 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:54772->10.242.241.18:8300: i/o timeout
2021/04/02 00:39:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:30 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:39:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:35 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:39:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:40 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:55024->10.242.241.18:8300: i/o timeout
2021/04/02 00:39:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:41 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:39:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:45 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:55096->10.242.241.18:8300: i/o timeout
2021/04/02 00:39:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:47 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:39:47 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:39:50 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:39:50 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:39:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:56 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:39:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:39:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:00 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:55342->10.242.241.18:8300: i/o timeout
2021/04/02 00:40:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:06 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:55418->10.242.241.18:8300: i/o timeout
2021/04/02 00:40:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:10 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:40:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:17 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:40:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:20 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:55642->10.242.241.18:8300: i/o timeout
2021/04/02 00:40:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:21 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:40:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:27 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:55724->10.242.241.18:8300: i/o timeout
2021/04/02 00:40:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:29 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:40:29 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:40:31 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:40:32 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:40:32 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/02 00:40:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:37 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:40:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:41 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:55938->10.242.241.18:8300: i/o timeout
2021/04/02 00:40:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:47 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:56030->10.242.241.18:8300: i/o timeout
2021/04/02 00:40:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:51 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:40:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:40:59 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:40:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:00 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:41:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:01 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:56232->10.242.241.18:8300: i/o timeout
2021/04/02 00:41:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:09 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:56342->10.242.241.18:8300: i/o timeout
2021/04/02 00:41:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:11 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:41:11 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:41:11 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:41:14 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:41:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:20 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:41:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:21 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:56544->10.242.241.18:8300: i/o timeout
2021/04/02 00:41:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:30 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:56662->10.242.241.18:8300: i/o timeout
2021/04/02 00:41:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:31 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:41:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:40 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:41:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:41 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:56842->10.242.241.18:8300: i/o timeout
2021/04/02 00:41:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:50 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:56972->10.242.241.18:8300: i/o timeout
2021/04/02 00:41:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:52 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:41:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:53 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:41:53 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:41:56 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:41:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:41:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:00 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:42:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:02 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:57136->10.242.241.18:8300: i/o timeout
2021/04/02 00:42:02 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:42:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:12 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:57294->10.242.241.18:8300: i/o timeout
2021/04/02 00:42:12 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:42:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:22 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:57436->10.242.241.18:8300: i/o timeout
2021/04/02 00:42:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:23 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:42:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:31 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:42:32 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:42:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:33 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:57602->10.242.241.18:8300: i/o timeout
2021/04/02 00:42:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:35 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:42:35 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:42:38 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:42:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:42 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:57738->10.242.241.18:8300: i/o timeout
2021/04/02 00:42:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:44 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:42:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:52 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:42:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:54 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:57916->10.242.241.18:8300: i/o timeout
2021/04/02 00:42:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:42:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:02 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:58032->10.242.241.18:8300: i/o timeout
2021/04/02 00:43:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:04 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:43:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:11 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:43:13 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:43:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:14 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:58212->10.242.241.18:8300: i/o timeout
2021/04/02 00:43:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:17 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:43:17 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:43:20 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:43:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:23 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:58330->10.242.241.18:8300: i/o timeout
2021/04/02 00:43:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:25 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:43:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:33 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:43:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:35 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:43:35 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:43:35 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:43:35 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:43:35 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/02 00:43:36 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/04/02 00:43:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:38 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500991053s
2021/04/02 00:43:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:54956->10.242.241.18:8301: i/o timeout
2021/04/02 00:43:40 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:43:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:42 [WARN] raft: Rejecting vote request from 10.242.241.18:8300 since we have a leader: 10.242.51.203:8300
2021/04/02 00:43:42 [ERR] raft: peer {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300} has newer term, stopping replication
2021/04/02 00:43:42 [INFO] raft: aborting pipeline replication to peer {Voter 89c1a268-ee85-dfa0-749a-f6802dc15803 10.242.189.8:8300}
2021/04/02 00:43:42 [INFO] consul: cluster leadership lost
2021/04/02 00:43:42 [INFO] raft: Node at 10.242.51.203:8300 [Follower] entering Follower state (Leader: "")
2021/04/02 00:43:43 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:43:43 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:43:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:55020->10.242.241.18:8301: i/o timeout
2021/04/02 00:43:44 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:43:49 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/02 00:43:49 [INFO] raft: Node at 10.242.51.203:8300 [Candidate] entering Candidate state in term 7
2021/04/02 00:43:49 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:43:49 [INFO] raft: Election won. Tally: 2
2021/04/02 00:43:49 [INFO] raft: Node at 10.242.51.203:8300 [Leader] entering Leader state
2021/04/02 00:43:49 [INFO] raft: Added peer 89c1a268-ee85-dfa0-749a-f6802dc15803, starting replication
2021/04/02 00:43:49 [INFO] raft: Added peer ab3e202d-9a1e-5268-25eb-4c4881c4e864, starting replication
2021/04/02 00:43:49 [INFO] consul: cluster leadership acquired
2021/04/02 00:43:49 [INFO] consul: New leader elected: dev-consul-server-2
2021/04/02 00:43:49 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:43:49 [INFO] raft: pipelining replication to peer {Voter 89c1a268-ee85-dfa0-749a-f6802dc15803 10.242.189.8:8300}
2021/04/02 00:43:49 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/02 00:43:50 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:43:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:48356->10.242.241.18:8302: i/o timeout
2021/04/02 00:43:51 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:43:52 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.501313745s
2021/04/02 00:43:52 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:43:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:54 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 4.951738822s
2021/04/02 00:43:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:56 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 7.403884678s
2021/04/02 00:43:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:43:59 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:43:59 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:43:59 [ERR] raft: Failed to make RequestVote RPC to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:58654->10.242.241.18:8300: i/o timeout
2021/04/02 00:43:59 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:58538->10.242.241.18:8300: i/o timeout
2021/04/02 00:43:59 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:44:00 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:58912->10.242.241.18:8300: i/o timeout
2021/04/02 00:44:00 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:44:00 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:44:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:48508->10.242.241.18:8302: i/o timeout
2021/04/02 00:44:01 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:44:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:09 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:59054->10.242.241.18:8300: i/o timeout
2021/04/02 00:44:09 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:44:10 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:59080->10.242.241.18:8300: i/o timeout
2021/04/02 00:44:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:48598->10.242.241.18:8302: i/o timeout
2021/04/02 00:44:11 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:44:11 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:44:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:15 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:44:15 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.241.18:48200
2021/04/02 00:44:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:19 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:59210->10.242.241.18:8300: i/o timeout
2021/04/02 00:44:19 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:44:21 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/02 00:44:21 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:44:21 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:44:21 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:59242->10.242.241.18:8300: i/o timeout
2021/04/02 00:44:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:22 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:44:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:48900->10.242.241.18:8302: i/o timeout
2021/04/02 00:44:26 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:44:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:29 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:44:29 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:44:29 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/02 00:44:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:29 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:59352->10.242.241.18:8300: i/o timeout
2021/04/02 00:44:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:32 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:59398->10.242.241.18:8300: i/o timeout
2021/04/02 00:44:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:55756->10.242.241.18:8301: i/o timeout
2021/04/02 00:44:33 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:44:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:35 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:44:35 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:44:35 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:44:35 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/02 00:44:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:39 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:44:39 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:44:39 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:59512->10.242.241.18:8300: i/o timeout
2021/04/02 00:44:39 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:44:42 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:44:43 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:59560->10.242.241.18:8300: i/o timeout
2021/04/02 00:44:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:43 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:44:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:49 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:59656->10.242.241.18:8300: i/o timeout
2021/04/02 00:44:49 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:44:51 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:44:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:53 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:59728->10.242.241.18:8300: i/o timeout
2021/04/02 00:44:54 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:44:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:44:59 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:59808->10.242.241.18:8300: i/o timeout
2021/04/02 00:45:00 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:45:00 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:45:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:04 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:59880->10.242.241.18:8300: i/o timeout
2021/04/02 00:45:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:05 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:45:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:10 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:59962->10.242.241.18:8300: i/o timeout
2021/04/02 00:45:10 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:45:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:15 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:60056->10.242.241.18:8300: i/o timeout
2021/04/02 00:45:17 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:45:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:20 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:60132->10.242.241.18:8300: i/o timeout
2021/04/02 00:45:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:21 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:45:21 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:45:22 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:45:24 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:45:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:27 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:60204->10.242.241.18:8300: i/o timeout
2021/04/02 00:45:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:29 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:45:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:32 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:60300->10.242.241.18:8300: i/o timeout
2021/04/02 00:45:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:34 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:45:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:39 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:60404->10.242.241.18:8300: i/o timeout
2021/04/02 00:45:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:42 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:45:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:44 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:60478->10.242.241.18:8300: i/o timeout
2021/04/02 00:45:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:49 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:45:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:52 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:60608->10.242.241.18:8300: i/o timeout
2021/04/02 00:45:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:58 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:45:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:45:59 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:60708->10.242.241.18:8300: i/o timeout
2021/04/02 00:46:00 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:46:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:03 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:46:03 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:46:06 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:46:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:08 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:60828->10.242.241.18:8300: i/o timeout
2021/04/02 00:46:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:10 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:46:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:19 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:46:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:20 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:32782->10.242.241.18:8300: i/o timeout
2021/04/02 00:46:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:29 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:32914->10.242.241.18:8300: i/o timeout
2021/04/02 00:46:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:30 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:46:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:40 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:46:40 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:33084->10.242.241.18:8300: i/o timeout
2021/04/02 00:46:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:45 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:46:45 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:46:48 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:46:48 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/02 00:46:48 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/02 00:46:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:50 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:33224->10.242.241.18:8300: i/o timeout
2021/04/02 00:46:50 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:46:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:46:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:47:00 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:47:00 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:33384->10.242.241.18:8300: i/o timeout
2021/04/02 00:47:01 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:47:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:47:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:47:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:47:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:47:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:47:10 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:47:11 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:33548->10.242.241.18:8300: i/o timeout
2021/04/02 00:47:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:47:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:47:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:47:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:47:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:47:20 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:33690->10.242.241.18:8300: i/o timeout
2021/04/02 00:47:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:47:22 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:47:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:47:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:47:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:47:27 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:47:27 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:47:30 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:47:31 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:47:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:47:32 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:33860->10.242.241.18:8300: i/o timeout
2021/04/02 00:47:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:47:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:47:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:47:38 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:47:38 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:47:38 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/02 00:47:38 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:47:38 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:47:38 [WARN] raft: Rejecting vote request from 10.242.241.18:8300 since we have a leader: 10.242.51.203:8300
2021/04/02 00:47:38 [ERR] raft: peer {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300} has newer term, stopping replication
2021/04/02 00:47:38 [INFO] raft: Node at 10.242.51.203:8300 [Follower] entering Follower state (Leader: "")
2021/04/02 00:47:38 [INFO] raft: aborting pipeline replication to peer {Voter 89c1a268-ee85-dfa0-749a-f6802dc15803 10.242.189.8:8300}
2021/04/02 00:47:38 [INFO] consul: cluster leadership lost
2021/04/02 00:47:38 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/02 00:47:46 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/04/02 00:47:46 [INFO] raft: Node at 10.242.51.203:8300 [Candidate] entering Candidate state in term 8
2021/04/02 00:47:46 [INFO] raft: Election won. Tally: 2
2021/04/02 00:47:46 [INFO] raft: Node at 10.242.51.203:8300 [Leader] entering Leader state
2021/04/02 00:47:46 [INFO] raft: Added peer 89c1a268-ee85-dfa0-749a-f6802dc15803, starting replication
2021/04/02 00:47:46 [INFO] raft: Added peer ab3e202d-9a1e-5268-25eb-4c4881c4e864, starting replication
2021/04/02 00:47:46 [INFO] consul: cluster leadership acquired
2021/04/02 00:47:46 [INFO] consul: New leader elected: dev-consul-server-2
2021/04/02 00:47:46 [INFO] raft: pipelining replication to peer {Voter 89c1a268-ee85-dfa0-749a-f6802dc15803 10.242.189.8:8300}
2021/04/02 00:47:46 [WARN] raft: AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300} rejected, sending older logs (next: 920)
2021/04/02 00:47:46 [INFO] raft: pipelining replication to peer {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}
2021/04/02 00:50:13 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:50:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:60848->10.242.241.18:8301: i/o timeout
2021/04/02 00:50:19 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:50:19 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:50:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:50:22 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500490108s
2021/04/02 00:50:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:60902->10.242.241.18:8301: i/o timeout
2021/04/02 00:50:22 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:50:22 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:50:22 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:50:22 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:50:22 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/02 00:50:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:50:24 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 4.917433053s
2021/04/02 00:50:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:50:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:50:27 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:50:27 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:50:27 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:50:27 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/02 00:50:29 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:50:29 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/02 00:50:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:50:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:32838->10.242.241.18:8301: i/o timeout
2021/04/02 00:50:34 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:50:41 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:50:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:50:47 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:50:47 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/02 00:50:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:50:48 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500179508s
2021/04/02 00:50:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:33086->10.242.241.18:8301: i/o timeout
2021/04/02 00:50:50 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:50:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:50:51 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 4.938062432s
2021/04/02 00:50:51 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:50:51 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:50:51 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/02 00:50:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:50:53 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:50:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:50:54 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:50:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:50:57 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:50:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:50:59 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500297284s
2021/04/02 00:51:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:51:01 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 4.983350965s
2021/04/02 00:51:02 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:51:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:51:03 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:51:03 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:51:03 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/02 00:51:03 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/02 00:51:05 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:51:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:33332->10.242.241.18:8301: i/o timeout
2021/04/02 00:51:06 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:51:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:51:06 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500279651s
2021/04/02 00:51:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:51:08 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:51:08 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:51:08 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:51:08 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/02 00:51:09 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:33358->10.242.241.18:8301: i/o timeout
2021/04/02 00:51:09 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:51:09 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 4.981589169s
2021/04/02 00:51:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:51:11 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 7.46292799s
2021/04/02 00:51:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:51:13 [ERR] raft: Failed to pipeline AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: write tcp 10.242.51.203:33986->10.242.241.18:8300: use of closed network connection
2021/04/02 00:51:13 [INFO] raft: aborting pipeline replication to peer {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}
2021/04/02 00:51:13 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:51:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:51:14 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:34218->10.242.241.18:8300: i/o timeout
2021/04/02 00:51:15 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:51:15 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.50021294s
2021/04/02 00:51:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:54952->10.242.241.18:8302: i/o timeout
2021/04/02 00:51:16 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:51:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:51:18 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 4.939485177s
2021/04/02 00:51:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:51:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:51:20 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 7.414382404s
2021/04/02 00:51:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:51:23 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:37238->10.242.241.18:8300: i/o timeout
2021/04/02 00:51:23 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:51:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:51:25 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:37276->10.242.241.18:8300: i/o timeout
2021/04/02 00:51:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:55112->10.242.241.18:8302: i/o timeout
2021/04/02 00:51:26 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:51:26 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:51:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:51:33 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:37394->10.242.241.18:8300: i/o timeout
2021/04/02 00:51:33 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:51:33 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:51:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:51:36 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured
2021/04/02 00:51:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:51:36 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:51:38 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500202869s
2021/04/02 00:51:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:51:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:51:40 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 4.944046736s
2021/04/02 00:51:42 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:51:42 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:51:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:51:43 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 7.420138701s
2021/04/02 00:51:43 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:37540->10.242.241.18:8300: i/o timeout
2021/04/02 00:51:43 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:51:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:51:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:55406->10.242.241.18:8302: i/o timeout
2021/04/02 00:51:46 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:51:46 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:37440->10.242.241.18:8300: i/o timeout
2021/04/02 00:51:47 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:51:47 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:51:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:51:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:51:52 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:51:52 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:51:52 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/02 00:51:52 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:51:52 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:51:52 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1)
2021/04/02 00:51:52 [WARN] raft: Rejecting vote request from 10.242.241.18:8300 since we have a leader: 10.242.51.203:8300
2021/04/02 00:51:52 [INFO] raft: pipelining replication to peer {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}
2021/04/02 00:51:59 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:52:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:34138->10.242.241.18:8301: i/o timeout
2021/04/02 00:52:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:34246->10.242.241.18:8301: i/o timeout
2021/04/02 00:52:07 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:52:07 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:52:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:54:19 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:54:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:54:21 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500228153s
2021/04/02 00:54:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:36230->10.242.241.18:8301: i/o timeout
2021/04/02 00:54:22 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:54:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:54:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:36280->10.242.241.18:8301: i/o timeout
2021/04/02 00:54:25 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:54:28 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:36322->10.242.241.18:8301: i/o timeout
2021/04/02 00:54:28 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:54:29 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:54:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:54:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:36370->10.242.241.18:8301: i/o timeout
2021/04/02 00:54:31 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:54:32 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500200194s
2021/04/02 00:54:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:54:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:54:34 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 4.999983954s
2021/04/02 00:54:35 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:36424->10.242.241.18:8301: i/o timeout
2021/04/02 00:54:35 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 00:54:35 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 00:54:35 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 00:54:35 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:54:35 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/02 00:54:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:57914->10.242.241.18:8302: i/o timeout
2021/04/02 00:54:36 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:54:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:54:37 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 7.437827223s
2021/04/02 00:54:37 [INFO] raft: aborting pipeline replication to peer {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}
2021/04/02 00:54:37 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:54:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:54:39 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.5003949s
2021/04/02 00:54:40 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:37754->10.242.241.18:8300: i/o timeout
2021/04/02 00:54:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:54:41 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:54:42 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 4.941328812s
2021/04/02 00:54:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:54:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:54:44 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 7.415350063s
2021/04/02 00:54:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:58074->10.242.241.18:8302: i/o timeout
2021/04/02 00:54:46 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:54:46 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:54:46 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:54:47 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:40260->10.242.241.18:8300: i/o timeout
2021/04/02 00:54:47 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:54:47 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:54:47 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/02 00:54:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:54:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:54:51 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:40332->10.242.241.18:8300: i/o timeout
2021/04/02 00:54:51 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:54:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:54:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:54:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:58222->10.242.241.18:8302: i/o timeout
2021/04/02 00:54:56 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:54:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:54:57 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:40420->10.242.241.18:8300: i/o timeout
2021/04/02 00:54:57 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:54:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:01 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:40486->10.242.241.18:8300: i/o timeout
2021/04/02 00:55:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:02 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:55:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:06 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/02 00:55:06 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:55:06 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:55:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:58292->10.242.241.18:8302: i/o timeout
2021/04/02 00:55:06 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 00:55:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:07 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:55:07 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.241.18:49782
2021/04/02 00:55:07 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:40568->10.242.241.18:8300: i/o timeout
2021/04/02 00:55:07 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:55:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:12 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:40660->10.242.241.18:8300: i/o timeout
2021/04/02 00:55:13 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:55:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:17 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:40728->10.242.241.18:8300: i/o timeout
2021/04/02 00:55:17 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:55:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:23 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:40812->10.242.241.18:8300: i/o timeout
2021/04/02 00:55:24 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:55:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:26 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:55:26 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:55:27 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:40866->10.242.241.18:8300: i/o timeout
2021/04/02 00:55:27 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:55:29 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:55:30 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:55:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:34 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:40982->10.242.241.18:8300: i/o timeout
2021/04/02 00:55:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:35 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:55:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:37 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:41030->10.242.241.18:8300: i/o timeout
2021/04/02 00:55:37 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:55:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:45 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:41144->10.242.241.18:8300: i/o timeout
2021/04/02 00:55:45 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:55:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:47 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:41180->10.242.241.18:8300: i/o timeout
2021/04/02 00:55:48 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:55:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:55 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:41306->10.242.241.18:8300: i/o timeout
2021/04/02 00:55:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:57 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:55:58 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:41334->10.242.241.18:8300: i/o timeout
2021/04/02 00:55:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:55:58 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:56:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:56:01 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 00:56:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:56:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:56:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:56:07 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:41460->10.242.241.18:8300: i/o timeout
2021/04/02 00:56:07 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 00:56:07 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 00:56:08 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:41490->10.242.241.18:8300: i/o timeout
2021/04/02 00:56:09 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:56:09 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 00:56:10 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:56:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:56:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:56:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:56:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:56:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:56:19 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:41654->10.242.241.18:8300: i/o timeout
2021/04/02 00:56:20 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:41662->10.242.241.18:8300: i/o timeout
2021/04/02 00:56:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:56:22 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:56:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:56:22 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:56:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:56:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:56:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:56:30 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 00:56:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:56:32 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:41840->10.242.241.18:8300: i/o timeout
2021/04/02 00:56:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:56:32 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:41844->10.242.241.18:8300: i/o timeout
2021/04/02 00:56:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:56:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:56:37 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:56:38 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 00:56:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:56:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:56:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 00:56:43 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.241.18
2021/04/02 00:56:43 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 00:56:43 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/02 00:56:43 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 00:56:43 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 00:56:43 [WARN] raft: Rejecting vote request from 10.242.241.18:8300 since we have a leader: 10.242.51.203:8300
2021/04/02 00:56:44 [INFO] raft: pipelining replication to peer {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}
2021/04/02 01:01:01 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 01:01:43 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 01:01:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:01:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:42794->10.242.241.18:8301: i/o timeout
2021/04/02 01:01:45 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 01:01:45 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500256045s
2021/04/02 01:01:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:01:49 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 01:01:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:02:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:43056->10.242.241.18:8301: i/o timeout
2021/04/02 01:02:02 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 01:02:02 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1)
2021/04/02 01:02:13 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 01:02:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:02:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:43256->10.242.241.18:8301: i/o timeout
2021/04/02 01:02:16 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received
2021/04/02 01:02:16 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500212101s
2021/04/02 01:02:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:02:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:02:19 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured
2021/04/02 01:02:19 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500227663s
2021/04/02 01:02:20 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/04/02 01:02:20 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.241.18
2021/04/02 01:02:20 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.241.18:8300) (DC: dc1)
2021/04/02 01:02:20 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/02 01:02:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:02:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:36566->10.242.241.18:8302: i/o timeout
2021/04/02 01:02:21 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 01:02:21 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 4.994220675s
2021/04/02 01:02:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:02:24 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 7.45498143s
2021/04/02 01:02:24 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:42072->10.242.241.18:8300: i/o timeout
2021/04/02 01:02:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:02:25 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:02:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:02:26 [INFO] raft: aborting pipeline replication to peer {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}
2021/04/02 01:02:26 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:02:26 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 01:02:26 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.241.18:57898
2021/04/02 01:02:26 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 01:02:26 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 01:02:29 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 2.500219194s
2021/04/02 01:02:29 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 01:02:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:02:31 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 4.961144106s
2021/04/02 01:02:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:02:34 [WARN] raft: Failed to contact ab3e202d-9a1e-5268-25eb-4c4881c4e864 in 7.382599132s
2021/04/02 01:02:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:02:35 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:47212->10.242.241.18:8300: i/o timeout
2021/04/02 01:02:35 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:02:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:36814->10.242.241.18:8302: i/o timeout
2021/04/02 01:02:36 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 01:02:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:02:36 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:47234->10.242.241.18:8300: i/o timeout
2021/04/02 01:02:36 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:02:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:02:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:02:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:02:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:02:45 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:47376->10.242.241.18:8300: i/o timeout
2021/04/02 01:02:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:36986->10.242.241.18:8302: i/o timeout
2021/04/02 01:02:46 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 01:02:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:02:46 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:02:46 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:47388->10.242.241.18:8300: i/o timeout
2021/04/02 01:02:46 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:02:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:02:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:02:51 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/02 01:02:51 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.241.18
2021/04/02 01:02:51 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/02 01:02:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:02:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:02:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.51.203:37132->10.242.241.18:8302: i/o timeout
2021/04/02 01:02:56 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received
2021/04/02 01:02:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:02:56 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:47554->10.242.241.18:8300: i/o timeout
2021/04/02 01:02:56 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:47558->10.242.241.18:8300: i/o timeout
2021/04/02 01:02:56 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:02:57 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:02:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:06 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:47704->10.242.241.18:8300: i/o timeout
2021/04/02 01:03:06 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:03:07 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:47716->10.242.241.18:8300: i/o timeout
2021/04/02 01:03:08 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:03:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:08 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 01:03:08 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 01:03:11 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 01:03:11 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 01:03:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:16 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:47868->10.242.241.18:8300: i/o timeout
2021/04/02 01:03:17 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:03:18 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:47890->10.242.241.18:8300: i/o timeout
2021/04/02 01:03:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:19 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:03:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:27 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:48008->10.242.241.18:8300: i/o timeout
2021/04/02 01:03:27 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:03:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:29 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:48046->10.242.241.18:8300: i/o timeout
2021/04/02 01:03:30 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:03:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:37 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:48184->10.242.241.18:8300: i/o timeout
2021/04/02 01:03:37 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:03:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:40 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:48226->10.242.241.18:8300: i/o timeout
2021/04/02 01:03:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:41 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:03:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:43 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 01:03:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:47 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:48332->10.242.241.18:8300: i/o timeout
2021/04/02 01:03:48 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:03:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:50 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 01:03:50 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 01:03:51 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:48402->10.242.241.18:8300: i/o timeout
2021/04/02 01:03:53 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:03:53 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 01:03:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:58 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:48508->10.242.241.18:8300: i/o timeout
2021/04/02 01:03:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:03:59 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:04:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:03 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:48584->10.242.241.18:8300: i/o timeout
2021/04/02 01:04:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:06 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:04:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:09 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:48688->10.242.241.18:8300: i/o timeout
2021/04/02 01:04:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:12 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:04:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:16 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:16 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:48806->10.242.241.18:8300: i/o timeout
2021/04/02 01:04:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:21 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 01:04:22 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:48886->10.242.241.18:8300: i/o timeout
2021/04/02 01:04:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:22 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:04:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:27 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:04:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:32 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 01:04:32 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 01:04:32 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:49052->10.242.241.18:8300: i/o timeout
2021/04/02 01:04:35 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 01:04:35 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/02 01:04:35 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/02 01:04:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:37 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:49124->10.242.241.18:8300: i/o timeout
2021/04/02 01:04:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:42 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:43 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:04:44 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:46 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:47 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:04:48 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:50 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:52 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:53 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:49366->10.242.241.18:8300: i/o timeout
2021/04/02 01:04:54 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:56 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:04:57 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:49420->10.242.241.18:8300: i/o timeout
2021/04/02 01:04:58 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:05:00 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:05:01 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.241.18:8301
2021/04/02 01:05:02 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:05:04 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:05:04 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:05:06 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:05:07 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:05:08 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:05:10 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:05:12 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:05:13 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.241.18:8302
2021/04/02 01:05:14 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:05:14 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/02 01:05:14 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF
2021/04/02 01:05:14 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:49676->10.242.241.18:8300: i/o timeout
2021/04/02 01:05:17 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/02 01:05:17 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:49728->10.242.241.18:8300: i/o timeout
2021/04/02 01:05:18 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:05:20 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:05:22 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:05:24 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:05:25 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:05:26 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:05:27 [WARN] Unable to get address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864, using fallback address 10.242.241.18:8300: Could not find address for server id ab3e202d-9a1e-5268-25eb-4c4881c4e864
2021/04/02 01:05:28 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:05:30 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:05:32 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:05:34 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:05:35 [ERR] raft: Failed to heartbeat to 10.242.241.18:8300: read tcp 10.242.51.203:50008->10.242.241.18:8300: i/o timeout
2021/04/02 01:05:36 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:05:37 [ERR] raft: Failed to AppendEntries to {Voter ab3e202d-9a1e-5268-25eb-4c4881c4e864 10.242.241.18:8300}: read tcp 10.242.51.203:50042->10.242.241.18:8300: i/o timeout
2021/04/02 01:05:38 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding
2021/04/02 01:05:40 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding