By type
+ CONSUL_BIND=
+ [ -n ]
+ CONSUL_CLIENT=
+ [ -n ]
+ CONSUL_DATA_DIR=/consul/data
+ CONSUL_CONFIG_DIR=/consul/config
+ [ -n ]
+ [ a = - ]
+ [ agent = agent ]
+ shift
+ set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
+ [ consul = consul ]
+ stat -c %u /consul/data
+ id -u consul
+ [ 100 != 100 ]
+ stat -c %u /consul/config
+ id -u consul
+ [ 100 != 100 ]
+ [ ! -z ]
+ exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui
bootstrap_expect > 0: expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
Version: 'v1.0.6'
Node ID: '7f153cf0-41de-1328-0bf9-c00e86bd10fe'
Node name: 'dev-consul-server-2'
Datacenter: 'dc1' (Segment: '')
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.242.195.202 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2021/04/07 07:47:26 [INFO] raft: Initial configuration (index=0): []
2021/04/07 07:47:26 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.195.202
2021/04/07 07:47:26 [INFO] raft: Node at 10.242.195.202:8300 [Follower] entering Follower state (Leader: "")
2021/04/07 07:47:26 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.195.202
2021/04/07 07:47:26 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.195.202:8300) (DC: dc1)
2021/04/07 07:47:26 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"
2021/04/07 07:47:26 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2021/04/07 07:47:26 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2021/04/07 07:47:26 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
2021/04/07 07:47:26 [INFO] agent: started state syncer
2021/04/07 07:47:26 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer
2021/04/07 07:47:26 [INFO] agent: Joining LAN cluster...
2021/04/07 07:47:26 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server]
2021/04/07 07:47:26 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.216.75
2021/04/07 07:47:26 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-ncbt6 10.242.146.140
2021/04/07 07:47:26 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.146.141
2021/04/07 07:47:26 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.216.75:8300) (DC: dc1)
2021/04/07 07:47:26 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.242.216.75:8300,10.242.146.141:8300,10.242.195.202:8300
2021/04/07 07:47:26 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.146.141
2021/04/07 07:47:26 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.146.141:8300) (DC: dc1)
2021/04/07 07:47:26 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.216.75
2021/04/07 07:47:26 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/04/07 07:47:26 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"
2021/04/07 07:47:26 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host
2021/04/07 07:47:26 [INFO] agent: (LAN) joined: 2 Err:
2021/04/07 07:47:26 [INFO] agent: Join LAN completed. Synced with 2 initial agents
2021/04/07 07:47:27 [DEBUG] raft-net: 10.242.195.202:8300 accepted connection from: 10.242.146.141:47742
2021/04/07 07:47:27 [INFO] agent: Synced node info
2021/04/07 07:47:27 [INFO] consul: New leader elected: dev-consul-server-0
2021/04/07 07:47:28 [DEBUG] raft-net: 10.242.195.202:8300 accepted connection from: 10.242.146.141:47754
==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=8b3b4cf9-77b4-80dd-5121-a5400e4afc6e&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2021/04/07 08:09:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:55476->10.242.146.140:8301: i/o timeout
2021/04/07 08:09:44 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:09:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:55752->10.242.146.140:8301: i/o timeout
2021/04/07 08:09:58 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:10:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:55944->10.242.146.140:8301: i/o timeout
2021/04/07 08:10:07 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:10:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:55980->10.242.146.140:8301: i/o timeout
2021/04/07 08:10:10 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:10:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:56060->10.242.146.140:8301: i/o timeout
2021/04/07 08:10:15 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:10:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:56096->10.242.146.140:8301: i/o timeout
2021/04/07 08:10:17 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:10:18 [INFO] memberlist: Marking dev-consul-7467bf7b46-ncbt6 as failed, suspect timeout reached (2 peer confirmations)
2021/04/07 08:10:18 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-ncbt6 10.242.146.140
2021/04/07 08:10:19 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/04/07 08:10:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:50016->10.242.146.141:8301: i/o timeout
2021/04/07 08:10:22 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:10:23 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/07 08:10:23 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.146.140:54136
2021/04/07 08:10:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:50052->10.242.146.141:8301: i/o timeout
2021/04/07 08:10:24 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:10:26 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 08:10:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:50112->10.242.146.141:8301: i/o timeout
2021/04/07 08:10:27 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:10:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:50140->10.242.146.141:8301: i/o timeout
2021/04/07 08:10:29 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:10:31 [WARN] raft: Heartbeat timeout from "10.242.146.141:8300" reached, starting election
2021/04/07 08:10:31 [INFO] raft: Node at 10.242.195.202:8300 [Candidate] entering Candidate state in term 3
2021/04/07 08:10:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:49302->10.242.146.141:8302: i/o timeout
2021/04/07 08:10:31 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/04/07 08:10:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:50164->10.242.146.141:8301: i/o timeout
2021/04/07 08:10:31 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:10:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:50182->10.242.146.141:8301: i/o timeout
2021/04/07 08:10:33 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:10:34 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations)
2021/04/07 08:10:34 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.146.141
2021/04/07 08:10:34 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.146.141:8300) (DC: dc1)
2021/04/07 08:10:39 [WARN] raft: Election timeout reached, restarting election
2021/04/07 08:10:39 [INFO] raft: Node at 10.242.195.202:8300 [Candidate] entering Candidate state in term 4
2021/04/07 08:10:39 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:10:39 [ERR] http: Request GET /v1/catalog/service/config_binding_service, error: No cluster leader from=10.242.195.204:59908
2021/04/07 08:10:41 [ERR] http: Request GET /v1/catalog/service/config-binding-service, error: No cluster leader from=10.242.195.234:42100
2021/04/07 08:10:41 [ERR] raft: Failed to make RequestVote RPC to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:41912->10.242.146.141:8300: i/o timeout
2021/04/07 08:10:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:49452->10.242.146.141:8302: i/o timeout
2021/04/07 08:10:41 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/04/07 08:10:44 [DEBUG] raft-net: 10.242.195.202:8300 accepted connection from: 10.242.216.75:50814
2021/04/07 08:10:47 [ERR] agent: Coordinate update error: No cluster leader
2021/04/07 08:10:47 [ERR] memberlist: failed to receive: read tcp 10.242.195.202:8301->10.242.146.140:56926: i/o timeout from=10.242.146.140:56926
2021/04/07 08:10:47 [ERR] memberlist: Failed to send error: write tcp 10.242.195.202:8301->10.242.146.140:56926: i/o timeout from=10.242.146.140:56926
2021/04/07 08:10:48 [WARN] raft: Election timeout reached, restarting election
2021/04/07 08:10:48 [INFO] raft: Node at 10.242.195.202:8300 [Candidate] entering Candidate state in term 5
2021/04/07 08:10:48 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:10:48 [INFO] raft: Election won. Tally: 2
2021/04/07 08:10:48 [INFO] raft: Node at 10.242.195.202:8300 [Leader] entering Leader state
2021/04/07 08:10:48 [INFO] raft: Added peer 03b716bd-ba23-5bb2-9001-c33bcc1dadf9, starting replication
2021/04/07 08:10:48 [INFO] raft: Added peer f012f32b-5d81-d4e0-d844-840e0764da2e, starting replication
2021/04/07 08:10:48 [INFO] consul: cluster leadership acquired
2021/04/07 08:10:48 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:10:48 [INFO] consul: New leader elected: dev-consul-server-2
2021/04/07 08:10:48 [INFO] raft: pipelining replication to peer {Voter 03b716bd-ba23-5bb2-9001-c33bcc1dadf9 10.242.216.75:8300}
2021/04/07 08:10:48 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/04/07 08:10:48 [INFO] consul: member 'dev-consul-7467bf7b46-ncbt6' failed, marking health critical
2021/04/07 08:10:49 [ERR] raft: Failed to make RequestVote RPC to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:42030->10.242.146.141:8300: i/o timeout
2021/04/07 08:10:49 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:10:49 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/04/07 08:10:50 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-ncbt6 10.242.146.140
2021/04/07 08:10:50 [INFO] consul: member 'dev-consul-7467bf7b46-ncbt6' joined, marking health alive
2021/04/07 08:10:50 [WARN] raft: Rejecting vote request from 10.242.146.141:8300 since we have a leader: 10.242.195.202:8300
2021/04/07 08:10:50 [INFO] raft: pipelining replication to peer {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}
2021/04/07 08:10:50 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:10:50 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.146.141
2021/04/07 08:10:50 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.146.141:8300) (DC: dc1)
2021/04/07 08:10:50 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/04/07 08:10:53 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:10:53 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/07 08:10:59 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:11:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:11:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:50644->10.242.146.141:8301: i/o timeout
2021/04/07 08:11:00 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:11:02 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 2.500199721s
2021/04/07 08:11:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:11:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:50672->10.242.146.141:8301: i/o timeout
2021/04/07 08:11:02 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:11:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:11:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:56852->10.242.146.140:8301: i/o timeout
2021/04/07 08:11:04 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:11:04 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 4.972474054s
2021/04/07 08:11:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:11:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:49882->10.242.146.141:8302: i/o timeout
2021/04/07 08:11:06 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/04/07 08:11:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:50732->10.242.146.141:8301: i/o timeout
2021/04/07 08:11:06 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:11:07 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 7.433110701s
2021/04/07 08:11:11 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:11:15 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:11:15 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/07 08:11:15 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/07 08:11:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:11:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:50908->10.242.146.141:8301: i/o timeout
2021/04/07 08:11:16 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:11:17 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 2.50034514s
2021/04/07 08:11:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:11:19 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:57098->10.242.146.140:8301: i/o timeout
2021/04/07 08:11:19 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:11:19 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 4.967988717s
2021/04/07 08:11:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:11:21 [WARN] memberlist: Was able to connect to dev-consul-server-0 but other probes failed, network may be misconfigured
2021/04/07 08:11:22 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.146.141
2021/04/07 08:11:22 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.146.141:8300) (DC: dc1)
2021/04/07 08:11:22 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/04/07 08:11:22 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/04/07 08:11:23 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:11:23 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.146.141
2021/04/07 08:11:23 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.146.141:8300) (DC: dc1)
2021/04/07 08:11:23 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/04/07 08:11:29 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:11:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:11:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:51136->10.242.146.141:8301: i/o timeout
2021/04/07 08:11:30 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:11:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:11:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:57308->10.242.146.140:8301: i/o timeout
2021/04/07 08:11:33 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:11:33 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 2.500240252s
2021/04/07 08:11:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:11:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:57386->10.242.146.140:8301: i/o timeout
2021/04/07 08:11:36 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:11:37 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 2.502308323s
2021/04/07 08:11:37 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:11:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:11:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:57432->10.242.146.140:8301: i/o timeout
2021/04/07 08:11:38 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:11:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:11:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:51310->10.242.146.141:8301: i/o timeout
2021/04/07 08:11:40 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:11:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:50462->10.242.146.141:8302: i/o timeout
2021/04/07 08:11:41 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/04/07 08:11:42 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-ncbt6 10.242.146.140
2021/04/07 08:11:42 [INFO] consul: member 'dev-consul-7467bf7b46-ncbt6' failed, marking health critical
2021/04/07 08:11:42 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations)
2021/04/07 08:11:42 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.146.141
2021/04/07 08:11:42 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.146.141:8300) (DC: dc1)
2021/04/07 08:11:42 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/04/07 08:11:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:11:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:57480->10.242.146.140:8301: i/o timeout
2021/04/07 08:11:42 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:11:43 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:11:44 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:11:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:11:44 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:11:45 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:11:46 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:11:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:11:47 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/07 08:11:47 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/04/07 08:11:47 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:11:48 [ERR] raft: Failed to pipeline AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: write tcp 10.242.195.202:42172->10.242.146.141:8300: use of closed network connection
2021/04/07 08:11:48 [INFO] raft: aborting pipeline replication to peer {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}
2021/04/07 08:11:48 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:11:48 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:11:48 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.146.141
2021/04/07 08:11:48 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.146.141:8300) (DC: dc1)
2021/04/07 08:11:48 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/04/07 08:11:49 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:11:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:11:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:51478->10.242.146.141:8301: i/o timeout
2021/04/07 08:11:51 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:11:52 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 2.500226727s
2021/04/07 08:11:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:11:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:51490->10.242.146.141:8301: i/o timeout
2021/04/07 08:11:53 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:11:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:11:55 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 4.916977113s
2021/04/07 08:11:55 [INFO] raft: pipelining replication to peer {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}
2021/04/07 08:11:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:11:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:51570->10.242.146.141:8301: i/o timeout
2021/04/07 08:11:56 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:11:58 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 2.507265111s
2021/04/07 08:11:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:11:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:51608->10.242.146.141:8301: i/o timeout
2021/04/07 08:11:58 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:12:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:12:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:51640->10.242.146.141:8301: i/o timeout
2021/04/07 08:12:00 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:12:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:12:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:51670->10.242.146.141:8301: i/o timeout
2021/04/07 08:12:02 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:12:03 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations)
2021/04/07 08:12:03 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.146.141
2021/04/07 08:12:03 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.146.141:8300) (DC: dc1)
2021/04/07 08:12:03 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/04/07 08:12:03 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:12:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:12:05 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.146.141
2021/04/07 08:12:05 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.146.141:8300) (DC: dc1)
2021/04/07 08:12:05 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/04/07 08:12:05 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-ncbt6 10.242.146.140
2021/04/07 08:12:05 [INFO] consul: member 'dev-consul-7467bf7b46-ncbt6' joined, marking health alive
2021/04/07 08:13:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:52714->10.242.146.141:8301: i/o timeout
2021/04/07 08:13:08 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:13:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:52972->10.242.146.141:8301: i/o timeout
2021/04/07 08:13:24 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:13:25 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:13:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:59226->10.242.146.140:8301: i/o timeout
2021/04/07 08:13:31 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:13:33 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:13:34 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 2.500228728s
2021/04/07 08:13:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:13:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:59280->10.242.146.140:8301: i/o timeout
2021/04/07 08:13:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:13:37 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:13:37 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/07 08:13:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:53152->10.242.146.141:8301: i/o timeout
2021/04/07 08:13:37 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:13:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:13:39 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 2.500391739s
2021/04/07 08:13:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:13:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:59348->10.242.146.140:8301: i/o timeout
2021/04/07 08:13:40 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:13:42 [INFO] raft: aborting pipeline replication to peer {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}
2021/04/07 08:13:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:13:43 [INFO] memberlist: Marking dev-consul-7467bf7b46-ncbt6 as failed, suspect timeout reached (1 peer confirmations)
2021/04/07 08:13:43 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-ncbt6 10.242.146.140
2021/04/07 08:13:43 [INFO] consul: member 'dev-consul-7467bf7b46-ncbt6' failed, marking health critical
2021/04/07 08:13:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:59440->10.242.146.140:8301: i/o timeout
2021/04/07 08:13:43 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:13:44 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 2.500246855s
2021/04/07 08:13:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:13:45 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/07 08:13:45 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.146.141:47734
2021/04/07 08:13:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:53298->10.242.146.141:8301: i/o timeout
2021/04/07 08:13:45 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:13:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:13:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:52456->10.242.146.141:8302: i/o timeout
2021/04/07 08:13:46 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/04/07 08:13:47 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 4.997666544s
2021/04/07 08:13:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:53344->10.242.146.141:8301: i/o timeout
2021/04/07 08:13:47 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:13:48 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:42196->10.242.146.141:8300: i/o timeout
2021/04/07 08:13:48 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.146.141
2021/04/07 08:13:48 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.146.141:8300) (DC: dc1)
2021/04/07 08:13:48 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/04/07 08:13:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:13:48 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:13:49 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 7.436339664s
2021/04/07 08:13:50 [ERR] memberlist: failed to receive: read tcp 10.242.195.202:8301->10.242.146.141:50124: i/o timeout from=10.242.146.141:50124
2021/04/07 08:13:50 [ERR] memberlist: Failed to send error: write tcp 10.242.195.202:8301->10.242.146.141:50124: i/o timeout from=10.242.146.141:50124
2021/04/07 08:13:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:13:52 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:44998->10.242.146.141:8300: i/o timeout
2021/04/07 08:13:52 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:13:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:13:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:13:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:13:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:52544->10.242.146.141:8302: i/o timeout
2021/04/07 08:13:56 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/04/07 08:13:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:13:58 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/07 08:13:58 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/04/07 08:13:58 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:45104->10.242.146.141:8300: i/o timeout
2021/04/07 08:13:59 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:14:01 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:14:01 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/07 08:14:02 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:45154->10.242.146.141:8300: i/o timeout
2021/04/07 08:14:02 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:14:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:52800->10.242.146.141:8302: i/o timeout
2021/04/07 08:14:06 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/04/07 08:14:06 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 08:14:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:09 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:45300->10.242.146.141:8300: i/o timeout
2021/04/07 08:14:10 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:14:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:12 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:45342->10.242.146.141:8300: i/o timeout
2021/04/07 08:14:12 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:14:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:14 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/07 08:14:14 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.146.141
2021/04/07 08:14:14 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/04/07 08:14:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:52946->10.242.146.141:8302: i/o timeout
2021/04/07 08:14:16 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/04/07 08:14:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:20 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:45460->10.242.146.141:8300: i/o timeout
2021/04/07 08:14:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:20 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/07 08:14:20 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.146.140:33098
2021/04/07 08:14:21 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:14:22 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:45488->10.242.146.141:8300: i/o timeout
2021/04/07 08:14:22 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:14:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:26 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.146.141:8302
2021/04/07 08:14:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:31 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:45652->10.242.146.141:8300: i/o timeout
2021/04/07 08:14:32 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:14:32 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:45676->10.242.146.141:8300: i/o timeout
2021/04/07 08:14:32 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:14:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:40 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/07 08:14:40 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/04/07 08:14:42 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:45876->10.242.146.141:8300: i/o timeout
2021/04/07 08:14:42 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:45886->10.242.146.141:8300: i/o timeout
2021/04/07 08:14:42 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:14:43 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:14:43 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:14:43 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded
2021/04/07 08:14:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:46 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.146.141:8301
2021/04/07 08:14:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:52 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:46144->10.242.146.141:8300: i/o timeout
2021/04/07 08:14:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:53 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:14:53 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:46168->10.242.146.141:8300: i/o timeout
2021/04/07 08:14:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:54 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:14:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:14:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:03 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:46422->10.242.146.141:8300: i/o timeout
2021/04/07 08:15:03 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:15:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:04 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:46452->10.242.146.141:8300: i/o timeout
2021/04/07 08:15:06 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:15:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:13 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:46656->10.242.146.141:8300: i/o timeout
2021/04/07 08:15:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:15 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:15:16 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:46698->10.242.146.141:8300: i/o timeout
2021/04/07 08:15:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:18 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:15:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:22 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/07 08:15:22 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/04/07 08:15:23 [ERR] memberlist: failed to receive: read tcp 10.242.195.202:8301->10.242.146.141:50372: i/o timeout from=10.242.146.141:50372
2021/04/07 08:15:23 [ERR] memberlist: Failed to send error: write tcp 10.242.195.202:8301->10.242.146.141:50372: i/o timeout from=10.242.146.141:50372
2021/04/07 08:15:25 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:46880->10.242.146.141:8300: i/o timeout
2021/04/07 08:15:25 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:15:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:26 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.146.141:8302
2021/04/07 08:15:26 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.146.141:8301
2021/04/07 08:15:27 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:15:28 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:46956->10.242.146.141:8300: i/o timeout
2021/04/07 08:15:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:31 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:15:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:37 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:47176->10.242.146.141:8300: i/o timeout
2021/04/07 08:15:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:41 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:47250->10.242.146.141:8300: i/o timeout
2021/04/07 08:15:42 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:15:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:46 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.146.141
2021/04/07 08:15:46 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0)
2021/04/07 08:15:46 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.146.141:8300) (DC: dc1)
2021/04/07 08:15:46 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/04/07 08:15:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:55860->10.242.146.141:8301: i/o timeout
2021/04/07 08:15:48 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:15:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:55902->10.242.146.141:8301: i/o timeout
2021/04/07 08:15:50 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:15:52 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (0 peer confirmations)
2021/04/07 08:15:52 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.146.141
2021/04/07 08:15:52 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.146.141:8300) (DC: dc1)
2021/04/07 08:15:52 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/04/07 08:15:52 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:47492->10.242.146.141:8300: i/o timeout
2021/04/07 08:15:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:55938->10.242.146.141:8301: i/o timeout
2021/04/07 08:15:52 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:15:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:15:57 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:47584->10.242.146.141:8300: i/o timeout
2021/04/07 08:15:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:02 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:16:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:04 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/07 08:16:04 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/04/07 08:16:06 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 08:16:07 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:16:08 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:16:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:12 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:47942->10.242.146.141:8300: i/o timeout
2021/04/07 08:16:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:18 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:48046->10.242.146.141:8300: i/o timeout
2021/04/07 08:16:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:23 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:16:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:26 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/07 08:16:26 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.146.141:47866
2021/04/07 08:16:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:26 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.146.141:8302
2021/04/07 08:16:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:29 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:16:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:33 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:48366->10.242.146.141:8300: i/o timeout
2021/04/07 08:16:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:39 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:48504->10.242.146.141:8300: i/o timeout
2021/04/07 08:16:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:43 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:16:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:46 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/07 08:16:46 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/04/07 08:16:46 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 08:16:49 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:16:50 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:16:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:53 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:48790->10.242.146.141:8300: i/o timeout
2021/04/07 08:16:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:16:59 [WARN] raft: Rejecting vote request from 10.242.146.141:8300 since we have a leader: 10.242.195.202:8300
2021/04/07 08:16:59 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.146.141
2021/04/07 08:16:59 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/04/07 08:17:00 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.146.141
2021/04/07 08:17:00 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.146.141:8300) (DC: dc1)
2021/04/07 08:17:00 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/04/07 08:17:00 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:48916->10.242.146.141:8300: i/o timeout
2021/04/07 08:17:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:57422->10.242.146.141:8301: i/o timeout
2021/04/07 08:17:01 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:17:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:57492->10.242.146.141:8301: i/o timeout
2021/04/07 08:17:03 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:17:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:05 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (0 peer confirmations)
2021/04/07 08:17:05 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.146.141
2021/04/07 08:17:05 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.146.141:8300) (DC: dc1)
2021/04/07 08:17:05 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/04/07 08:17:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:57522->10.242.146.141:8301: i/o timeout
2021/04/07 08:17:06 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:17:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:11 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:17:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:13 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:49236->10.242.146.141:8300: i/o timeout
2021/04/07 08:17:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:56884->10.242.146.141:8302: i/o timeout
2021/04/07 08:17:16 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/04/07 08:17:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:21 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/07 08:17:21 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.146.140:35538
2021/04/07 08:17:21 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:49390->10.242.146.141:8300: i/o timeout
2021/04/07 08:17:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:23 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:17:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:57098->10.242.146.141:8302: i/o timeout
2021/04/07 08:17:26 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/04/07 08:17:26 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 08:17:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:28 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/07 08:17:28 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/04/07 08:17:31 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:17:32 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:17:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:33 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:49672->10.242.146.141:8300: i/o timeout
2021/04/07 08:17:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:57318->10.242.146.141:8302: i/o timeout
2021/04/07 08:17:36 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/04/07 08:17:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:42 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:49834->10.242.146.141:8300: i/o timeout
2021/04/07 08:17:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:44 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:17:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:45 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.146.141
2021/04/07 08:17:45 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/04/07 08:17:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:57418->10.242.146.141:8302: i/o timeout
2021/04/07 08:17:46 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/04/07 08:17:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:53 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:17:54 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:50096->10.242.146.141:8300: i/o timeout
2021/04/07 08:17:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:17:56 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.146.141:8302
2021/04/07 08:17:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:03 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:50278->10.242.146.141:8300: i/o timeout
2021/04/07 08:18:04 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:18:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:06 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 08:18:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:10 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/07 08:18:10 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/04/07 08:18:13 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:18:14 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:18:14 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:50508->10.242.146.141:8300: i/o timeout
2021/04/07 08:18:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:24 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:50720->10.242.146.141:8300: i/o timeout
2021/04/07 08:18:24 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:18:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:34 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:50938->10.242.146.141:8300: i/o timeout
2021/04/07 08:18:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:34 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:18:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:37 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.146.141
2021/04/07 08:18:37 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.146.141:8300) (DC: dc1)
2021/04/07 08:18:37 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/04/07 08:18:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:39 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-ncbt6 10.242.146.140
2021/04/07 08:18:39 [INFO] consul: member 'dev-consul-7467bf7b46-ncbt6' joined, marking health alive
2021/04/07 08:18:39 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.146.141
2021/04/07 08:18:39 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/04/07 08:18:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:59548->10.242.146.141:8301: i/o timeout
2021/04/07 08:18:39 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:18:39 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 2.50027255s
2021/04/07 08:18:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:42 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 4.938450792s
2021/04/07 08:18:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:42 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:59586->10.242.146.141:8301: i/o timeout
2021/04/07 08:18:42 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:18:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:44 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 7.358935202s
2021/04/07 08:18:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:37564->10.242.146.140:8301: i/o timeout
2021/04/07 08:18:45 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:18:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:46 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:58818->10.242.146.141:8302: i/o timeout
2021/04/07 08:18:46 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/04/07 08:18:48 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:51142->10.242.146.141:8300: i/o timeout
2021/04/07 08:18:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:59724->10.242.146.141:8301: i/o timeout
2021/04/07 08:18:48 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:18:48 [ERR] memberlist: failed to receive: read tcp 10.242.195.202:8301->10.242.146.140:32948: i/o timeout from=10.242.146.140:32948
2021/04/07 08:18:48 [ERR] memberlist: Failed to send error: write tcp 10.242.195.202:8301->10.242.146.140:32948: i/o timeout from=10.242.146.140:32948
2021/04/07 08:18:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:59754->10.242.146.141:8301: i/o timeout
2021/04/07 08:18:50 [INFO] memberlist: Suspect dev-consul-server-0 has failed, no acks received
2021/04/07 08:18:51 [INFO] memberlist: Marking dev-consul-server-0 as failed, suspect timeout reached (1 peer confirmations)
2021/04/07 08:18:51 [INFO] serf: EventMemberFailed: dev-consul-server-0 10.242.146.141
2021/04/07 08:18:51 [INFO] consul: Removing LAN server dev-consul-server-0 (Addr: tcp/10.242.146.141:8300) (DC: dc1)
2021/04/07 08:18:51 [INFO] consul: member 'dev-consul-server-0' failed, marking health critical
2021/04/07 08:18:52 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/07 08:18:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:52 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/04/07 08:18:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:37744->10.242.146.140:8301: i/o timeout
2021/04/07 08:18:52 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:18:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:37788->10.242.146.140:8301: i/o timeout
2021/04/07 08:18:54 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:18:54 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:51412->10.242.146.141:8300: i/o timeout
2021/04/07 08:18:55 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:18:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:56 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:59050->10.242.146.141:8302: i/o timeout
2021/04/07 08:18:56 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/04/07 08:18:57 [INFO] memberlist: Marking dev-consul-7467bf7b46-ncbt6 as failed, suspect timeout reached (1 peer confirmations)
2021/04/07 08:18:57 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-ncbt6 10.242.146.140
2021/04/07 08:18:57 [INFO] consul: member 'dev-consul-7467bf7b46-ncbt6' failed, marking health critical
2021/04/07 08:18:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:37876->10.242.146.140:8301: i/o timeout
2021/04/07 08:18:57 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:18:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:18:58 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:51480->10.242.146.141:8300: i/o timeout
2021/04/07 08:18:59 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:19:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:05 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:19:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:59274->10.242.146.141:8302: i/o timeout
2021/04/07 08:19:06 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/04/07 08:19:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:09 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:51748->10.242.146.141:8300: i/o timeout
2021/04/07 08:19:10 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:19:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:15 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:51864->10.242.146.141:8300: i/o timeout
2021/04/07 08:19:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:59500->10.242.146.141:8302: i/o timeout
2021/04/07 08:19:16 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/04/07 08:19:16 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/07 08:19:16 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.146.141
2021/04/07 08:19:16 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/04/07 08:19:16 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.146.141:8301
2021/04/07 08:19:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:20 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:51980->10.242.146.141:8300: i/o timeout
2021/04/07 08:19:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:21 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:19:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:22 [ERR] memberlist: failed to receive: read tcp 10.242.195.202:8301->10.242.146.141:50718: i/o timeout from=10.242.146.141:50718
2021/04/07 08:19:22 [ERR] memberlist: Failed to send error: write tcp 10.242.195.202:8301->10.242.146.141:50718: i/o timeout from=10.242.146.141:50718
2021/04/07 08:19:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:25 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:19:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:31 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:52222->10.242.146.141:8300: i/o timeout
2021/04/07 08:19:32 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:19:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:34 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/07 08:19:34 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/04/07 08:19:35 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:52298->10.242.146.141:8300: i/o timeout
2021/04/07 08:19:37 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:19:38 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-lbqch 10.242.213.154
2021/04/07 08:19:38 [INFO] consul: member 'dev-consul-7467bf7b46-lbqch' joined, marking health alive
2021/04/07 08:19:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:42 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:52464->10.242.146.141:8300: i/o timeout
2021/04/07 08:19:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:43 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:19:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:45 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:19:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:48 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.146.141
2021/04/07 08:19:48 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-0.dc1)
2021/04/07 08:19:48 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/04/07 08:19:48 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:19:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:49 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 2.500381225s
2021/04/07 08:19:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:52 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 4.958437212s
2021/04/07 08:19:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:54 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 7.358340764s
2021/04/07 08:19:55 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:52788->10.242.146.141:8300: i/o timeout
2021/04/07 08:19:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:56 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 08:19:58 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:52746->10.242.146.141:8300: i/o timeout
2021/04/07 08:19:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:19:59 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:20:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:60422->10.242.146.141:8302: i/o timeout
2021/04/07 08:20:01 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/04/07 08:20:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:04 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:20:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:05 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:20:05 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:20:06 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:20:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:08 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 2.50039243s
2021/04/07 08:20:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:10 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 4.983439423s
2021/04/07 08:20:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:60706->10.242.146.141:8302: i/o timeout
2021/04/07 08:20:11 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/04/07 08:20:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:12 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 7.427224763s
2021/04/07 08:20:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:15 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:53140->10.242.146.141:8300: i/o timeout
2021/04/07 08:20:16 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:53318->10.242.146.141:8300: i/o timeout
2021/04/07 08:20:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:16 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/07 08:20:16 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/04/07 08:20:16 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:20:19 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:20:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:32954->10.242.146.141:8302: i/o timeout
2021/04/07 08:20:21 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/04/07 08:20:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:24 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/07 08:20:24 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.146.141
2021/04/07 08:20:24 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/04/07 08:20:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:26 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:20:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:26 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:53658->10.242.146.141:8300: i/o timeout
2021/04/07 08:20:27 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:20:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:36 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:53992->10.242.146.141:8300: i/o timeout
2021/04/07 08:20:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:36 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 08:20:37 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:54076->10.242.146.141:8300: i/o timeout
2021/04/07 08:20:38 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:20:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:42 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/07 08:20:42 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.146.141:48662
2021/04/07 08:20:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:46 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:20:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:48 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-ncbt6 10.242.146.140
2021/04/07 08:20:48 [INFO] consul: member 'dev-consul-7467bf7b46-ncbt6' joined, marking health alive
2021/04/07 08:20:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:49 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:20:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:50 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 2.500198502s
2021/04/07 08:20:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:41462->10.242.146.140:8301: i/o timeout
2021/04/07 08:20:51 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:20:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:53 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 4.960908881s
2021/04/07 08:20:54 [INFO] memberlist: Marking dev-consul-7467bf7b46-ncbt6 as failed, suspect timeout reached (2 peer confirmations)
2021/04/07 08:20:54 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-ncbt6 10.242.146.140
2021/04/07 08:20:54 [INFO] consul: member 'dev-consul-7467bf7b46-ncbt6' failed, marking health critical
2021/04/07 08:20:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:41580->10.242.146.140:8301: i/o timeout
2021/04/07 08:20:54 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:20:55 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 7.420062948s
2021/04/07 08:20:56 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:54834->10.242.146.141:8300: i/o timeout
2021/04/07 08:20:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:56 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.146.141:8302
2021/04/07 08:20:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:20:58 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/07 08:20:58 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/04/07 08:20:59 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:54442->10.242.146.141:8300: i/o timeout
2021/04/07 08:21:00 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:21:01 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:21:01 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/07 08:21:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:06 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:21:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:09 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.146.141
2021/04/07 08:21:09 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/04/07 08:21:10 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:55614->10.242.146.141:8300: i/o timeout
2021/04/07 08:21:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:11 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:21:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:16 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:55858->10.242.146.141:8300: i/o timeout
2021/04/07 08:21:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:21 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:56026->10.242.146.141:8300: i/o timeout
2021/04/07 08:21:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:35566->10.242.146.141:8302: i/o timeout
2021/04/07 08:21:21 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/04/07 08:21:21 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:21:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:26 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:21:28 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/07 08:21:28 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.146.140:36732
2021/04/07 08:21:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:30 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:35798->10.242.146.141:8302: i/o timeout
2021/04/07 08:21:31 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/04/07 08:21:31 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:56516->10.242.146.141:8300: i/o timeout
2021/04/07 08:21:32 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:21:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:36 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:56696->10.242.146.141:8300: i/o timeout
2021/04/07 08:21:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:39 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-ncbt6 10.242.146.140
2021/04/07 08:21:39 [INFO] consul: member 'dev-consul-7467bf7b46-ncbt6' joined, marking health alive
2021/04/07 08:21:40 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.216.75
2021/04/07 08:21:40 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.216.75:8300) (DC: dc1)
2021/04/07 08:21:40 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical
2021/04/07 08:21:40 [WARN] Unable to get address for server id 03b716bd-ba23-5bb2-9001-c33bcc1dadf9, using fallback address 10.242.216.75:8300: Could not find address for server id 03b716bd-ba23-5bb2-9001-c33bcc1dadf9
2021/04/07 08:21:40 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.216.75
2021/04/07 08:21:40 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.216.75:8300) (DC: dc1)
2021/04/07 08:21:40 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive
2021/04/07 08:21:40 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:21:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:42 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 2.50238775s
2021/04/07 08:21:42 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:43510->10.242.146.140:8301: i/o timeout
2021/04/07 08:21:43 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:21:44 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 4.962721229s
2021/04/07 08:21:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:43570->10.242.146.140:8301: i/o timeout
2021/04/07 08:21:45 [INFO] memberlist: Suspect dev-consul-7467bf7b46-ncbt6 has failed, no acks received
2021/04/07 08:21:46 [INFO] memberlist: Marking dev-consul-7467bf7b46-ncbt6 as failed, suspect timeout reached (2 peer confirmations)
2021/04/07 08:21:46 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-ncbt6 10.242.146.140
2021/04/07 08:21:46 [INFO] consul: member 'dev-consul-7467bf7b46-ncbt6' failed, marking health critical
2021/04/07 08:21:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:46 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 08:21:47 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:21:47 [WARN] raft: Failed to contact f012f32b-5d81-d4e0-d844-840e0764da2e in 7.459140122s
2021/04/07 08:21:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:49 [ERR] memberlist: failed to receive: read tcp 10.242.195.202:8301->10.242.146.140:34372: i/o timeout from=10.242.146.140:34372
2021/04/07 08:21:49 [ERR] memberlist: Failed to send error: write tcp 10.242.195.202:8301->10.242.146.140:34372: i/o timeout from=10.242.146.140:34372
2021/04/07 08:21:50 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:56918->10.242.146.141:8300: i/o timeout
2021/04/07 08:21:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:51 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:21:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:36766->10.242.146.141:8302: i/o timeout
2021/04/07 08:21:51 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/04/07 08:21:51 [INFO] memberlist: Marking dev-consul-server-0.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/04/07 08:21:51 [INFO] serf: EventMemberFailed: dev-consul-server-0.dc1 10.242.146.141
2021/04/07 08:21:51 [INFO] consul: Handled member-failed event for server "dev-consul-server-0.dc1" in area "wan"
2021/04/07 08:21:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:54 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:21:57 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: read tcp 10.242.195.202:57464->10.242.146.141:8300: i/o timeout
2021/04/07 08:21:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:01 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:57636->10.242.146.141:8300: i/o timeout
2021/04/07 08:22:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.195.202:36962->10.242.146.141:8302: i/o timeout
2021/04/07 08:22:01 [INFO] memberlist: Suspect dev-consul-server-0.dc1 has failed, no acks received
2021/04/07 08:22:02 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:22:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:07 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:22:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:12 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: read tcp 10.242.195.202:58090->10.242.146.141:8300: i/o timeout
2021/04/07 08:22:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:12 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:22:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:17 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:22:17 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/07 08:22:17 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error making call: EOF
2021/04/07 08:22:19 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:22:20 [ERR] yamux: keepalive failed: i/o deadline reached
2021/04/07 08:22:20 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.146.141:49128
2021/04/07 08:22:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:22 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:22:23 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:22:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:26 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.146.141:8302
2021/04/07 08:22:26 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 08:22:27 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:22:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:28 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:22:31 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:22:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:33 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:22:34 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:22:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:37 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:22:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:40 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:22:43 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:22:43 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/07 08:22:44 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:22:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:45 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:22:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:47 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:22:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:52 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:22:55 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:22:55 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:22:56 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:22:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:22:57 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:22:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:04 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:04 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:23:06 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:23:06 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 08:23:07 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:23:08 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:23:08 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:23:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:16 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:16 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:23:18 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:23:18 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:23:19 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:23:19 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:23:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:28 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:23:28 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:28 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:23:29 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:23:31 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:23:31 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/07 08:23:32 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:33 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:23:34 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:36 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:38 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:23:38 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:40 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:40 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:23:43 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:23:43 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:23:44 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:46 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:46 [INFO] serf: attempting reconnect to dev-consul-server-0 10.242.146.141:8301
2021/04/07 08:23:48 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:23:48 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:48 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:23:50 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:52 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:52 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:23:55 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:23:55 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/07 08:23:56 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:56 [INFO] serf: attempting reconnect to dev-consul-server-0.dc1 10.242.146.141:8302
2021/04/07 08:23:58 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:23:58 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:23:58 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:24:00 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:24:02 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:24:04 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:24:05 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:24:06 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:24:08 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:24:08 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:24:10 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:24:10 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:24:12 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:24:14 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:24:14 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:24:17 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:24:17 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded
2021/04/07 08:24:18 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:24:18 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:24:20 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:24:20 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:24:22 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:24:24 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:24:26 [WARN] consul: error getting server health from "dev-consul-server-0": last request still outstanding
2021/04/07 08:24:26 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:24:26 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 08:24:29 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:24:29 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.146.190
2021/04/07 08:24:29 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan"
2021/04/07 08:24:29 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.146.190
2021/04/07 08:24:29 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.146.190:8300) (DC: dc1)
2021/04/07 08:24:29 [INFO] raft: Updating configuration with AddNonvoter (f81e4fe0-3b6c-d1f7-c30b-453e56927a96, 10.242.146.190:8300) to [{Suffrage:Voter ID:03b716bd-ba23-5bb2-9001-c33bcc1dadf9 Address:10.242.216.75:8300} {Suffrage:Voter ID:f012f32b-5d81-d4e0-d844-840e0764da2e Address:10.242.146.141:8300} {Suffrage:Voter ID:7f153cf0-41de-1328-0bf9-c00e86bd10fe Address:10.242.195.202:8300} {Suffrage:Nonvoter ID:f81e4fe0-3b6c-d1f7-c30b-453e56927a96 Address:10.242.146.190:8300}]
2021/04/07 08:24:29 [INFO] raft: Added peer f81e4fe0-3b6c-d1f7-c30b-453e56927a96, starting replication
2021/04/07 08:24:29 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive
2021/04/07 08:24:29 [WARN] raft: AppendEntries to {Nonvoter f81e4fe0-3b6c-d1f7-c30b-453e56927a96 10.242.146.190:8300} rejected, sending older logs (next: 1)
2021/04/07 08:24:29 [INFO] raft: pipelining replication to peer {Nonvoter f81e4fe0-3b6c-d1f7-c30b-453e56927a96 10.242.146.190:8300}
2021/04/07 08:24:29 [WARN] consul: error getting server health from "dev-consul-server-0": context deadline exceeded
2021/04/07 08:24:31 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:24:38 [WARN] consul: error getting server health from "dev-consul-server-0": rpc error getting client: failed to get conn: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:24:39 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:24:41 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:24:48 [INFO] autopilot: Promoting Server (ID: "f81e4fe0-3b6c-d1f7-c30b-453e56927a96" Address: "10.242.146.190:8300") to voter
2021/04/07 08:24:48 [INFO] raft: Updating configuration with AddStaging (f81e4fe0-3b6c-d1f7-c30b-453e56927a96, 10.242.146.190:8300) to [{Suffrage:Voter ID:03b716bd-ba23-5bb2-9001-c33bcc1dadf9 Address:10.242.216.75:8300} {Suffrage:Voter ID:f012f32b-5d81-d4e0-d844-840e0764da2e Address:10.242.146.141:8300} {Suffrage:Voter ID:7f153cf0-41de-1328-0bf9-c00e86bd10fe Address:10.242.195.202:8300} {Suffrage:Voter ID:f81e4fe0-3b6c-d1f7-c30b-453e56927a96 Address:10.242.146.190:8300}]
2021/04/07 08:24:48 [INFO] autopilot: Attempting removal of stale Server (ID: "f012f32b-5d81-d4e0-d844-840e0764da2e" Address: "10.242.146.141:8300")
2021/04/07 08:24:48 [INFO] raft: Updating configuration with RemoveServer (f012f32b-5d81-d4e0-d844-840e0764da2e, ) to [{Suffrage:Voter ID:03b716bd-ba23-5bb2-9001-c33bcc1dadf9 Address:10.242.216.75:8300} {Suffrage:Voter ID:7f153cf0-41de-1328-0bf9-c00e86bd10fe Address:10.242.195.202:8300} {Suffrage:Voter ID:f81e4fe0-3b6c-d1f7-c30b-453e56927a96 Address:10.242.146.190:8300}]
2021/04/07 08:24:48 [INFO] raft: Removed peer f012f32b-5d81-d4e0-d844-840e0764da2e, stopping replication after 688
2021/04/07 08:24:49 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:24:52 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:24:59 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:25:02 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:25:06 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 08:25:09 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:25:13 [WARN] Unable to get address for server id f012f32b-5d81-d4e0-d844-840e0764da2e, using fallback address 10.242.146.141:8300: Could not find address for server id f012f32b-5d81-d4e0-d844-840e0764da2e
2021/04/07 08:25:19 [ERR] raft: Failed to AppendEntries to {Voter f012f32b-5d81-d4e0-d844-840e0764da2e 10.242.146.141:8300}: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:25:23 [ERR] raft: Failed to heartbeat to 10.242.146.141:8300: dial tcp ->10.242.146.141:8300: i/o timeout
2021/04/07 08:26:46 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 08:28:26 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 08:29:36 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 08:30:46 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 08:32:26 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 08:36:06 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 08:36:46 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 08:39:26 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 08:41:06 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 08:42:46 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 08:56:56 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 09:02:36 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 09:05:46 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 09:10:26 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 09:13:06 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 09:16:46 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 09:18:26 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 09:19:36 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 09:23:46 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301
2021/04/07 09:24:56 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-ncbt6 10.242.146.140:8301