By type
+ CONSUL_BIND=
+ [ -n ]
+ CONSUL_CLIENT=
+ [ -n ]
+ CONSUL_DATA_DIR=/consul/data
+ CONSUL_CONFIG_DIR=/consul/config
+ [ -n ]
+ [ a = - ]
+ [ agent = agent ]
+ shift
+ set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=onap-consul-server-0.consul-server -retry-join=onap-consul-server-1.consul-server -retry-join=onap-consul-server-2.consul-server -client=0.0.0.0 -server -ui
+ [ consul = consul ]
+ stat -c %u /consul/data
+ id -u consul
+ [ 100 != 100 ]
+ stat -c %u /consul/config
+ id -u consul
+ [ 100 != 100 ]
+ [ ! -z ]
+ exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=onap-consul-server-0.consul-server -retry-join=onap-consul-server-1.consul-server -retry-join=onap-consul-server-2.consul-server -client=0.0.0.0 -server -ui
bootstrap_expect > 0: expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
Version: 'v1.0.6'
Node ID: '99ed9889-91d3-a4dc-1078-c8e74f10faee'
Node name: 'onap-consul-server-0'
Datacenter: 'dc1' (Segment: '')
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.233.69.31 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
==> Log data will now stream in as it occurs:
2021/06/04 09:44:47 [INFO] raft: Initial configuration (index=0): []
2021/06/04 09:44:47 [INFO] raft: Node at 10.233.69.31:8300 [Follower] entering Follower state (Leader: "")
2021/06/04 09:44:47 [INFO] serf: EventMemberJoin: onap-consul-server-0.dc1 10.233.69.31
2021/06/04 09:44:47 [INFO] serf: EventMemberJoin: onap-consul-server-0 10.233.69.31
2021/06/04 09:44:47 [INFO] consul: Handled member-join event for server "onap-consul-server-0.dc1" in area "wan"
2021/06/04 09:44:47 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2021/06/04 09:44:47 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp) 2021/06/04 09:44:47 [INFO] agent: (LAN) joining: [onap-consul-server-0.consul-server onap-consul-server-1.consul-server onap-consul-server-2.consul-server]
2021/06/04 09:44:47 [INFO] consul: Adding LAN server onap-consul-server-0 (Addr: tcp/10.233.69.31:8300) (DC: dc1)
2021/06/04 09:44:47 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
2021/06/04 09:44:47 [INFO] agent: started state syncer
2021/06/04 09:44:47 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer
2021/06/04 09:44:47 [INFO] agent: Joining LAN cluster...
2021/06/04 09:44:49 [WARN] memberlist: Failed to resolve onap-consul-server-0.consul-server: lookup onap-consul-server-0.consul-server on 169.254.25.10:53: no such host
2021/06/04 09:44:52 [WARN] memberlist: Failed to resolve onap-consul-server-1.consul-server: lookup onap-consul-server-1.consul-server on 169.254.25.10:53: no such host
2021/06/04 09:44:54 [WARN] memberlist: Failed to resolve onap-consul-server-2.consul-server: lookup onap-consul-server-2.consul-server on 169.254.25.10:53: no such host
2021/06/04 09:44:54 [INFO] agent: (LAN) joined: 0 Err: 3 error(s) occurred:
* Failed to resolve onap-consul-server-0.consul-server: lookup onap-consul-server-0.consul-server on 169.254.25.10:53: no such host
* Failed to resolve onap-consul-server-1.consul-server: lookup onap-consul-server-1.consul-server on 169.254.25.10:53: no such host
* Failed to resolve onap-consul-server-2.consul-server: lookup onap-consul-server-2.consul-server on 169.254.25.10:53: no such host
2021/06/04 09:44:54 [WARN] agent: Join LAN failed: , retrying in 30s
2021/06/04 09:44:55 [ERR] agent: failed to sync remote state: No cluster leader
2021/06/04 09:44:56 [WARN] raft: no known peers, aborting election
2021/06/04 09:45:10 [ERR] agent: Coordinate update error: No cluster leader
==> Newer Consul version available: 1.9.5 (currently running: 1.0.6)
2021/06/04 09:45:24 [INFO] agent: (LAN) joining: [onap-consul-server-0.consul-server onap-consul-server-1.consul-server onap-consul-server-2.consul-server]
2021/06/04 09:45:25 [ERR] agent: failed to sync remote state: No cluster leader
2021/06/04 09:45:28 [WARN] memberlist: Failed to resolve onap-consul-server-1.consul-server: lookup onap-consul-server-1.consul-server on 169.254.25.10:53: no such host
2021/06/04 09:45:30 [WARN] memberlist: Failed to resolve onap-consul-server-2.consul-server: lookup onap-consul-server-2.consul-server on 169.254.25.10:53: no such host
2021/06/04 09:45:30 [INFO] agent: (LAN) joined: 1 Err:
2021/06/04 09:45:30 [INFO] agent: Join LAN completed. Synced with 1 initial agents
2021/06/04 09:45:41 [ERR] agent: Coordinate update error: No cluster leader
2021/06/04 09:45:42 [INFO] serf: EventMemberJoin: onap-consul-5bbdf6b858-w749h 10.233.72.75
2021/06/04 09:45:54 [ERR] agent: failed to sync remote state: No cluster leader
2021/06/04 09:46:07 [ERR] agent: Coordinate update error: No cluster leader
2021/06/04 09:46:31 [ERR] agent: failed to sync remote state: No cluster leader
2021/06/04 09:46:41 [ERR] agent: Coordinate update error: No cluster leader
2021/06/04 09:46:55 [ERR] agent: failed to sync remote state: No cluster leader
2021/06/04 09:47:08 [INFO] serf: EventMemberJoin: onap-consul-server-1 10.233.74.2
2021/06/04 09:47:08 [INFO] consul: Adding LAN server onap-consul-server-1 (Addr: tcp/10.233.74.2:8300) (DC: dc1)
2021/06/04 09:47:08 [INFO] serf: EventMemberJoin: onap-consul-server-1.dc1 10.233.74.2
2021/06/04 09:47:08 [INFO] consul: Handled member-join event for server "onap-consul-server-1.dc1" in area "wan"
2021/06/04 09:47:11 [ERR] agent: Coordinate update error: No cluster leader
2021/06/04 09:47:23 [ERR] agent: failed to sync remote state: No cluster leader
2021/06/04 09:47:38 [ERR] agent: Coordinate update error: No cluster leader
2021/06/04 09:47:58 [ERR] agent: failed to sync remote state: No cluster leader
2021/06/04 09:48:09 [ERR] agent: Coordinate update error: No cluster leader
2021/06/04 09:48:33 [ERR] agent: failed to sync remote state: No cluster leader
2021/06/04 09:48:36 [ERR] agent: Coordinate update error: No cluster leader
2021/06/04 09:49:01 [ERR] agent: failed to sync remote state: No cluster leader
2021/06/04 09:49:13 [ERR] agent: Coordinate update error: No cluster leader
2021/06/04 09:49:37 [ERR] agent: failed to sync remote state: No cluster leader
2021/06/04 09:49:39 [ERR] agent: Coordinate update error: No cluster leader
2021/06/04 09:50:09 [INFO] serf: EventMemberJoin: onap-consul-server-2 10.233.77.160
2021/06/04 09:50:09 [INFO] consul: Adding LAN server onap-consul-server-2 (Addr: tcp/10.233.77.160:8300) (DC: dc1)
2021/06/04 09:50:09 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.233.74.2:8300,10.233.77.160:8300,10.233.69.31:8300
2021/06/04 09:50:10 [ERR] agent: Coordinate update error: No cluster leader
2021/06/04 09:50:11 [INFO] serf: EventMemberJoin: onap-consul-server-2.dc1 10.233.77.160
2021/06/04 09:50:11 [INFO] consul: Handled member-join event for server "onap-consul-server-2.dc1" in area "wan"
2021/06/04 09:50:12 [WARN] raft: Heartbeat timeout from "" reached, starting election
2021/06/04 09:50:12 [INFO] raft: Node at 10.233.69.31:8300 [Candidate] entering Candidate state in term 2
2021/06/04 09:50:13 [INFO] raft: Election won. Tally: 2
2021/06/04 09:50:13 [INFO] raft: Node at 10.233.69.31:8300 [Leader] entering Leader state
2021/06/04 09:50:13 [INFO] raft: Added peer 92efc7b2-20b6-f9b6-9717-be53b30b788e, starting replication
2021/06/04 09:50:13 [INFO] raft: Added peer b74fcffa-56a0-4a75-76db-1be2eb68a68c, starting replication
2021/06/04 09:50:13 [INFO] consul: cluster leadership acquired
2021/06/04 09:50:13 [INFO] consul: New leader elected: onap-consul-server-0
2021/06/04 09:50:13 [WARN] raft: AppendEntries to {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300} rejected, sending older logs (next: 1)
2021/06/04 09:50:13 [WARN] raft: AppendEntries to {Voter b74fcffa-56a0-4a75-76db-1be2eb68a68c 10.233.77.160:8300} rejected, sending older logs (next: 1)
2021/06/04 09:50:14 [INFO] raft: pipelining replication to peer {Voter b74fcffa-56a0-4a75-76db-1be2eb68a68c 10.233.77.160:8300}
2021/06/04 09:50:14 [INFO] raft: pipelining replication to peer {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}
2021/06/04 09:50:15 [INFO] agent: Synced node info
2021/06/04 09:50:17 [INFO] consul: member 'onap-consul-server-0' joined, marking health alive
2021/06/04 09:50:17 [INFO] consul: member 'onap-consul-5bbdf6b858-w749h' joined, marking health alive
2021/06/04 09:50:19 [INFO] consul: member 'onap-consul-server-1' joined, marking health alive
2021/06/04 09:50:20 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 09:50:21 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:50:21 [INFO] consul: member 'onap-consul-server-2' joined, marking health alive
2021/06/04 09:50:22 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 09:50:23 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:50:23 [WARN] consul: error getting server health from "onap-consul-server-2": last request still outstanding
2021/06/04 09:50:26 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 09:50:27 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:50:29 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:50:58 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 09:50:58 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 09:50:58 [WARN] consul: error getting server health from "onap-consul-server-0": context deadline exceeded
2021/06/04 09:51:00 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 09:51:00 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 09:51:04 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 09:51:05 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:51:07 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:51:22 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 09:51:22 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 09:51:23 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:51:25 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:51:27 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:51:29 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:51:29 [INFO] raft: aborting pipeline replication to peer {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}
2021/06/04 09:51:31 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:51:33 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:51:35 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:51:37 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:51:39 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:51:39 [ERR] raft: Failed to AppendEntries to {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}: read tcp 10.233.69.31:38664->10.233.74.2:8300: i/o timeout
2021/06/04 09:51:41 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:51:43 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:51:45 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:51:47 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:51:48 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 09:51:49 [WARN] consul: error getting server health from "onap-consul-server-2": last request still outstanding
2021/06/04 09:51:49 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:51:49 [ERR] raft: Failed to AppendEntries to {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}: read tcp 10.233.69.31:39148->10.233.74.2:8300: i/o timeout
2021/06/04 09:51:51 [WARN] consul: error getting server health from "onap-consul-server-2": last request still outstanding
2021/06/04 09:51:51 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:51:53 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:51:55 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:51:57 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:51:59 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:51:59 [ERR] raft: Failed to AppendEntries to {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}: read tcp 10.233.69.31:39214->10.233.74.2:8300: i/o timeout
2021/06/04 09:52:01 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:03 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:05 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:07 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:09 [ERR] raft: Failed to AppendEntries to {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}: read tcp 10.233.69.31:39334->10.233.74.2:8300: i/o timeout
2021/06/04 09:52:10 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 09:52:10 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 09:52:11 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:13 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:15 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:17 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:19 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:19 [ERR] raft: Failed to AppendEntries to {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}: read tcp 10.233.69.31:39272->10.233.74.2:8300: i/o timeout
2021/06/04 09:52:21 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:23 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:25 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:26 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 09:52:27 [WARN] consul: error getting server health from "onap-consul-server-2": last request still outstanding
2021/06/04 09:52:27 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:29 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:29 [WARN] consul: error getting server health from "onap-consul-server-2": last request still outstanding
2021/06/04 09:52:30 [ERR] raft: Failed to AppendEntries to {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}: read tcp 10.233.69.31:39402->10.233.74.2:8300: i/o timeout
2021/06/04 09:52:31 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:32 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 09:52:32 [WARN] consul: error getting server health from "onap-consul-server-0": context deadline exceeded
2021/06/04 09:52:33 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:33 [WARN] consul: error getting server health from "onap-consul-server-2": last request still outstanding
2021/06/04 09:52:35 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:37 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:39 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:40 [ERR] raft: Failed to AppendEntries to {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}: read tcp 10.233.69.31:39472->10.233.74.2:8300: i/o timeout
2021/06/04 09:52:41 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:43 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:45 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:47 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:49 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:50 [ERR] raft: Failed to AppendEntries to {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}: read tcp 10.233.69.31:39536->10.233.74.2:8300: i/o timeout
2021/06/04 09:52:51 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:53 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:55 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:57 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:52:59 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:53:01 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:53:01 [ERR] raft: Failed to AppendEntries to {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}: read tcp 10.233.69.31:39600->10.233.74.2:8300: i/o timeout
2021/06/04 09:53:03 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:53:05 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:53:07 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:53:09 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:53:11 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:53:12 [ERR] raft: Failed to AppendEntries to {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}: read tcp 10.233.69.31:39668->10.233.74.2:8300: i/o timeout
2021/06/04 09:53:13 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:53:15 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:53:17 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:53:19 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:53:24 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 09:53:24 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 09:53:25 [ERR] raft: Failed to AppendEntries to {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}: read tcp 10.233.69.31:39990->10.233.74.2:8300: i/o timeout
2021/06/04 09:53:28 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 09:53:28 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 09:53:30 [INFO] raft: pipelining replication to peer {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}
2021/06/04 09:53:46 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 09:53:47 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:53:49 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:53:51 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:53:53 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:53:53 [ERR] memberlist: Failed fallback ping: read tcp 10.233.69.31:51024->10.233.74.2:8301: i/o timeout
2021/06/04 09:53:53 [INFO] memberlist: Suspect onap-consul-server-1 has failed, no acks received
2021/06/04 09:53:53 [WARN] raft: Failed to contact 92efc7b2-20b6-f9b6-9717-be53b30b788e in 2.500211125s
2021/06/04 09:53:55 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:53:55 [INFO] raft: aborting pipeline replication to peer {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}
2021/06/04 09:53:57 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:53:57 [WARN] raft: Failed to contact 92efc7b2-20b6-f9b6-9717-be53b30b788e in 2.500299893s
2021/06/04 09:53:57 [ERR] memberlist: Failed fallback ping: read tcp 10.233.69.31:57248->10.233.74.2:8302: i/o timeout
2021/06/04 09:53:57 [INFO] memberlist: Suspect onap-consul-server-1.dc1 has failed, no acks received
2021/06/04 09:53:57 [INFO] memberlist: Marking onap-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/06/04 09:53:57 [INFO] serf: EventMemberFailed: onap-consul-server-1 10.233.74.2
2021/06/04 09:53:57 [INFO] consul: Removing LAN server onap-consul-server-1 (Addr: tcp/10.233.74.2:8300) (DC: dc1)
2021/06/04 09:53:57 [INFO] consul: member 'onap-consul-server-1' failed, marking health critical
2021/06/04 09:53:58 [WARN] Unable to get address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e, using fallback address 10.233.74.2:8300: Could not find address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e
2021/06/04 09:53:58 [ERR] memberlist: Failed fallback ping: read tcp 10.233.69.31:51062->10.233.74.2:8301: i/o timeout
2021/06/04 09:53:58 [INFO] memberlist: Suspect onap-consul-server-1 has failed, no acks received
2021/06/04 09:53:59 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:00 [WARN] raft: Failed to contact 92efc7b2-20b6-f9b6-9717-be53b30b788e in 2.500369913s
2021/06/04 09:54:01 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:03 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:03 [WARN] raft: Failed to contact 92efc7b2-20b6-f9b6-9717-be53b30b788e in 4.987209628s
2021/06/04 09:54:05 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:05 [ERR] raft: Failed to AppendEntries to {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}: read tcp 10.233.69.31:40340->10.233.74.2:8300: i/o timeout
2021/06/04 09:54:05 [WARN] Unable to get address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e, using fallback address 10.233.74.2:8300: Could not find address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e
2021/06/04 09:54:05 [WARN] raft: Failed to contact 92efc7b2-20b6-f9b6-9717-be53b30b788e in 7.462306189s
2021/06/04 09:54:07 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:08 [ERR] raft: Failed to heartbeat to 10.233.74.2:8300: read tcp 10.233.69.31:40178->10.233.74.2:8300: i/o timeout
2021/06/04 09:54:09 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:09 [WARN] Unable to get address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e, using fallback address 10.233.74.2:8300: Could not find address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e
2021/06/04 09:54:11 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:12 [ERR] memberlist: Failed fallback ping: read tcp 10.233.69.31:57342->10.233.74.2:8302: i/o timeout
2021/06/04 09:54:12 [INFO] memberlist: Suspect onap-consul-server-1.dc1 has failed, no acks received
2021/06/04 09:54:13 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:15 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:15 [ERR] raft: Failed to AppendEntries to {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}: read tcp 10.233.69.31:40408->10.233.74.2:8300: i/o timeout
2021/06/04 09:54:15 [WARN] Unable to get address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e, using fallback address 10.233.74.2:8300: Could not find address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e
2021/06/04 09:54:17 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:19 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:19 [ERR] raft: Failed to heartbeat to 10.233.74.2:8300: read tcp 10.233.69.31:40426->10.233.74.2:8300: i/o timeout
2021/06/04 09:54:19 [ERR] yamux: keepalive failed: i/o deadline reached
2021/06/04 09:54:19 [ERR] yamux: keepalive failed: i/o deadline reached
2021/06/04 09:54:19 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.233.74.2:58370
2021/06/04 09:54:19 [WARN] consul: error getting server health from "onap-consul-server-1": rpc error making call: EOF
2021/06/04 09:54:20 [WARN] Unable to get address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e, using fallback address 10.233.74.2:8300: Could not find address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e
2021/06/04 09:54:22 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 09:54:22 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 09:54:23 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:25 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:25 [ERR] raft: Failed to AppendEntries to {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}: read tcp 10.233.69.31:40474->10.233.74.2:8300: i/o timeout
2021/06/04 09:54:25 [WARN] Unable to get address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e, using fallback address 10.233.74.2:8300: Could not find address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e
2021/06/04 09:54:27 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:27 [INFO] memberlist: Marking onap-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/06/04 09:54:27 [INFO] serf: EventMemberFailed: onap-consul-server-1.dc1 10.233.74.2
2021/06/04 09:54:27 [INFO] consul: Handled member-failed event for server "onap-consul-server-1.dc1" in area "wan"
2021/06/04 09:54:27 [ERR] memberlist: Failed fallback ping: read tcp 10.233.69.31:57446->10.233.74.2:8302: i/o timeout
2021/06/04 09:54:27 [INFO] memberlist: Suspect onap-consul-server-1.dc1 has failed, no acks received
2021/06/04 09:54:29 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:30 [ERR] raft: Failed to heartbeat to 10.233.74.2:8300: read tcp 10.233.69.31:40500->10.233.74.2:8300: i/o timeout
2021/06/04 09:54:30 [WARN] Unable to get address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e, using fallback address 10.233.74.2:8300: Could not find address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e
2021/06/04 09:54:31 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:33 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:35 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:35 [ERR] raft: Failed to AppendEntries to {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}: read tcp 10.233.69.31:40540->10.233.74.2:8300: i/o timeout
2021/06/04 09:54:35 [WARN] Unable to get address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e, using fallback address 10.233.74.2:8300: Could not find address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e
2021/06/04 09:54:37 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:39 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:40 [ERR] raft: Failed to heartbeat to 10.233.74.2:8300: read tcp 10.233.69.31:40574->10.233.74.2:8300: i/o timeout
2021/06/04 09:54:41 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:41 [WARN] Unable to get address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e, using fallback address 10.233.74.2:8300: Could not find address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e
2021/06/04 09:54:43 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:45 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:45 [ERR] raft: Failed to AppendEntries to {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}: read tcp 10.233.69.31:40608->10.233.74.2:8300: i/o timeout
2021/06/04 09:54:45 [WARN] Unable to get address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e, using fallback address 10.233.74.2:8300: Could not find address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e
2021/06/04 09:54:47 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:49 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:51 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:51 [ERR] raft: Failed to heartbeat to 10.233.74.2:8300: read tcp 10.233.69.31:40644->10.233.74.2:8300: i/o timeout
2021/06/04 09:54:52 [WARN] Unable to get address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e, using fallback address 10.233.74.2:8300: Could not find address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e
2021/06/04 09:54:53 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:55 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:55 [ERR] raft: Failed to AppendEntries to {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}: read tcp 10.233.69.31:40676->10.233.74.2:8300: i/o timeout
2021/06/04 09:54:55 [WARN] Unable to get address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e, using fallback address 10.233.74.2:8300: Could not find address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e
2021/06/04 09:54:57 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:54:59 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:01 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:01 [ERR] yamux: keepalive failed: i/o deadline reached
2021/06/04 09:55:01 [WARN] consul: error getting server health from "onap-consul-server-1": rpc error making call: EOF
2021/06/04 09:55:02 [ERR] raft: Failed to heartbeat to 10.233.74.2:8300: read tcp 10.233.69.31:40716->10.233.74.2:8300: i/o timeout
2021/06/04 09:55:03 [WARN] Unable to get address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e, using fallback address 10.233.74.2:8300: Could not find address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e
2021/06/04 09:55:04 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 09:55:04 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 09:55:05 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:05 [ERR] raft: Failed to AppendEntries to {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}: read tcp 10.233.69.31:40738->10.233.74.2:8300: i/o timeout
2021/06/04 09:55:05 [WARN] Unable to get address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e, using fallback address 10.233.74.2:8300: Could not find address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e
2021/06/04 09:55:07 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:09 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:11 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:13 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:13 [ERR] raft: Failed to heartbeat to 10.233.74.2:8300: read tcp 10.233.69.31:40800->10.233.74.2:8300: i/o timeout
2021/06/04 09:55:14 [WARN] Unable to get address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e, using fallback address 10.233.74.2:8300: Could not find address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e
2021/06/04 09:55:15 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:15 [ERR] raft: Failed to AppendEntries to {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}: read tcp 10.233.69.31:40814->10.233.74.2:8300: i/o timeout
2021/06/04 09:55:16 [WARN] Unable to get address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e, using fallback address 10.233.74.2:8300: Could not find address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e
2021/06/04 09:55:17 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:17 [INFO] serf: attempting reconnect to onap-consul-server-1 10.233.74.2:8301
2021/06/04 09:55:19 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:21 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:23 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:24 [ERR] raft: Failed to heartbeat to 10.233.74.2:8300: read tcp 10.233.69.31:40864->10.233.74.2:8300: i/o timeout
2021/06/04 09:55:25 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:26 [WARN] Unable to get address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e, using fallback address 10.233.74.2:8300: Could not find address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e
2021/06/04 09:55:26 [ERR] raft: Failed to AppendEntries to {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}: read tcp 10.233.69.31:40882->10.233.74.2:8300: i/o timeout
2021/06/04 09:55:27 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:27 [WARN] Unable to get address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e, using fallback address 10.233.74.2:8300: Could not find address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e
2021/06/04 09:55:29 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:31 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:33 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:35 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:36 [ERR] raft: Failed to heartbeat to 10.233.74.2:8300: read tcp 10.233.69.31:40942->10.233.74.2:8300: i/o timeout
2021/06/04 09:55:37 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:37 [ERR] raft: Failed to AppendEntries to {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}: read tcp 10.233.69.31:40954->10.233.74.2:8300: i/o timeout
2021/06/04 09:55:38 [WARN] Unable to get address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e, using fallback address 10.233.74.2:8300: Could not find address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e
2021/06/04 09:55:39 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:40 [WARN] Unable to get address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e, using fallback address 10.233.74.2:8300: Could not find address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e
2021/06/04 09:55:41 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:43 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:43 [ERR] yamux: keepalive failed: i/o deadline reached
2021/06/04 09:55:43 [WARN] consul: error getting server health from "onap-consul-server-1": rpc error making call: EOF
2021/06/04 09:55:46 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 09:55:46 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 09:55:47 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:47 [INFO] serf: attempting reconnect to onap-consul-server-1.dc1 10.233.74.2:8302
2021/06/04 09:55:48 [ERR] raft: Failed to heartbeat to 10.233.74.2:8300: read tcp 10.233.69.31:41018->10.233.74.2:8300: i/o timeout
2021/06/04 09:55:49 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:50 [ERR] raft: Failed to AppendEntries to {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}: read tcp 10.233.69.31:41030->10.233.74.2:8300: i/o timeout
2021/06/04 09:55:51 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:51 [WARN] Unable to get address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e, using fallback address 10.233.74.2:8300: Could not find address for server id 92efc7b2-20b6-f9b6-9717-be53b30b788e
2021/06/04 09:55:52 [INFO] serf: EventMemberJoin: onap-consul-server-1.dc1 10.233.74.2
2021/06/04 09:55:52 [INFO] consul: Handled member-join event for server "onap-consul-server-1.dc1" in area "wan"
2021/06/04 09:55:52 [INFO] serf: EventMemberJoin: onap-consul-server-1 10.233.74.2
2021/06/04 09:55:52 [INFO] consul: Adding LAN server onap-consul-server-1 (Addr: tcp/10.233.74.2:8300) (DC: dc1)
2021/06/04 09:55:52 [INFO] consul: member 'onap-consul-server-1' joined, marking health alive
2021/06/04 09:55:53 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:54 [ERR] memberlist: Failed fallback ping: read tcp 10.233.69.31:51836->10.233.74.2:8301: i/o timeout
2021/06/04 09:55:54 [INFO] memberlist: Suspect onap-consul-server-1 has failed, no acks received
2021/06/04 09:55:55 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:57 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:55:59 [INFO] raft: pipelining replication to peer {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}
2021/06/04 09:56:36 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 09:56:36 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 09:56:36 [WARN] consul: error getting server health from "onap-consul-server-0": context deadline exceeded
2021/06/04 09:56:37 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:56:39 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:56:41 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:57:14 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 09:57:14 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 09:58:06 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 09:58:06 [WARN] consul: error getting server health from "onap-consul-server-0": context deadline exceeded
2021/06/04 09:58:07 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:58:09 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:58:11 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:58:13 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:58:40 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 09:58:58 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 09:59:22 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 09:59:23 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:59:25 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:59:27 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:59:28 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 09:59:28 [WARN] consul: error getting server health from "onap-consul-server-0": context deadline exceeded
2021/06/04 09:59:29 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 09:59:56 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 09:59:56 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 10:00:13 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 10:00:13 [WARN] consul: error getting server health from "onap-consul-server-2": last request still outstanding
2021/06/04 10:00:15 [WARN] consul: error getting server health from "onap-consul-server-2": last request still outstanding
2021/06/04 10:00:20 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 10:00:21 [WARN] consul: error getting server health from "onap-consul-server-2": last request still outstanding
2021/06/04 10:00:23 [WARN] consul: error getting server health from "onap-consul-server-2": last request still outstanding
2021/06/04 10:01:16 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 10:01:16 [WARN] consul: error getting server health from "onap-consul-server-0": context deadline exceeded
2021/06/04 10:01:20 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 10:01:21 [WARN] consul: error getting server health from "onap-consul-server-2": last request still outstanding
2021/06/04 10:01:23 [WARN] consul: error getting server health from "onap-consul-server-2": last request still outstanding
2021/06/04 10:03:36 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 10:03:36 [WARN] consul: error getting server health from "onap-consul-server-0": context deadline exceeded
2021/06/04 10:03:36 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 10:03:37 [WARN] consul: error getting server health from "onap-consul-server-2": last request still outstanding
2021/06/04 10:05:50 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 10:05:50 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 10:05:51 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 10:07:02 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 10:07:02 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 10:07:03 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 10:07:36 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 10:07:37 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 10:07:39 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 10:08:32 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 10:08:32 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 10:08:33 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 10:08:48 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 10:09:46 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 10:10:28 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 10:10:28 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 10:10:36 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 10:10:36 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 10:10:37 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 10:10:39 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 10:10:41 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 10:10:43 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 10:15:50 [ERR] memberlist: Failed fallback ping: write tcp 10.233.69.31:32964->10.233.74.2:8301: i/o timeout
2021/06/04 10:15:50 [WARN] memberlist: Refuting a suspect message (from: onap-consul-server-1)
2021/06/04 10:16:46 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 10:16:46 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 10:16:46 [WARN] consul: error getting server health from "onap-consul-server-0": context deadline exceeded
2021/06/04 10:16:46 [ERR] memberlist: Failed fallback ping: write tcp 10.233.69.31:33378->10.233.74.2:8301: i/o timeout
2021/06/04 10:16:46 [INFO] memberlist: Suspect onap-consul-server-1 has failed, no acks received
2021/06/04 10:16:46 [WARN] memberlist: Refuting a suspect message (from: onap-consul-5bbdf6b858-w749h)
2021/06/04 10:16:50 [WARN] memberlist: Refuting a suspect message (from: onap-consul-server-2)
2021/06/04 10:16:50 [ERR] memberlist: Failed fallback ping: write tcp 10.233.69.31:33402->10.233.77.160:8301: i/o timeout
2021/06/04 10:16:50 [INFO] memberlist: Suspect onap-consul-server-2 has failed, no acks received
2021/06/04 10:16:55 [WARN] raft: Failed to contact b74fcffa-56a0-4a75-76db-1be2eb68a68c in 5.191378754s
2021/06/04 10:16:55 [WARN] memberlist: Refuting a suspect message (from: onap-consul-5bbdf6b858-w749h)
2021/06/04 10:16:56 [WARN] consul: error getting server health from "onap-consul-server-1": context deadline exceeded
2021/06/04 10:16:56 [WARN] consul: error getting server health from "onap-consul-server-2": context deadline exceeded
2021/06/04 10:16:57 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 10:16:57 [WARN] consul: error getting server health from "onap-consul-server-2": last request still outstanding
2021/06/04 10:17:00 [WARN] raft: Failed to contact 92efc7b2-20b6-f9b6-9717-be53b30b788e in 4.966108219s
2021/06/04 10:17:00 [WARN] raft: Failed to contact b74fcffa-56a0-4a75-76db-1be2eb68a68c in 4.75360365s
2021/06/04 10:17:00 [WARN] raft: Failed to contact quorum of nodes, stepping down
2021/06/04 10:17:00 [INFO] raft: Node at 10.233.69.31:8300 [Follower] entering Follower state (Leader: "")
2021/06/04 10:17:00 [INFO] raft: aborting pipeline replication to peer {Voter 92efc7b2-20b6-f9b6-9717-be53b30b788e 10.233.74.2:8300}
2021/06/04 10:17:00 [INFO] raft: aborting pipeline replication to peer {Voter b74fcffa-56a0-4a75-76db-1be2eb68a68c 10.233.77.160:8300}
2021/06/04 10:17:00 [WARN] consul: error getting server health from "onap-consul-server-1": last request still outstanding
2021/06/04 10:17:00 [WARN] consul: error getting server health from "onap-consul-server-2": last request still outstanding
2021/06/04 10:17:00 [WARN] consul.coordinate: Batch update failed: node is not the leader
2021/06/04 10:17:01 [ERR] memberlist: Failed fallback ping: write tcp 10.233.69.31:42596->10.233.72.75:8301: i/o timeout
2021/06/04 10:17:01 [INFO] memberlist: Suspect onap-consul-5bbdf6b858-w749h has failed, no acks received
2021/06/04 10:17:02 [WARN] consul: error getting server health from "onap-consul-server-0": context deadline exceeded
2021/06/04 10:17:02 [ERR] memberlist: Failed fallback ping: write tcp 10.233.69.31:44812->10.233.77.160:8302: i/o timeout
2021/06/04 10:17:02 [INFO] memberlist: Suspect onap-consul-server-2.dc1 has failed, no acks received
2021/06/04 10:17:02 [INFO] consul: cluster leadership lost
2021/06/04 10:17:02 [WARN] memberlist: Refuting a suspect message (from: onap-consul-server-1.dc1)
2021/06/04 10:17:02 [DEBUG] raft-net: 10.233.69.31:8300 accepted connection from: 10.233.77.160:33452
2021/06/04 10:17:26 [WARN] yamux: failed to send ping reply: session shutdown
2021/06/04 10:17:26 [DEBUG] raft-net: 10.233.69.31:8300 accepted connection from: 10.233.74.2:43952
2021/06/04 10:17:26 [DEBUG] raft-net: 10.233.69.31:8300 accepted connection from: 10.233.77.160:33586
2021/06/04 10:17:26 [DEBUG] raft-net: 10.233.69.31:8300 accepted connection from: 10.233.77.160:33852
2021/06/04 10:17:26 [DEBUG] raft-net: 10.233.69.31:8300 accepted connection from: 10.233.77.160:33862
2021/06/04 10:17:26 [WARN] memberlist: Refuting a suspect message (from: onap-consul-server-1)
2021/06/04 10:17:26 [WARN] memberlist: Refuting a suspect message (from: onap-consul-server-0.dc1)
2021/06/04 10:17:26 [WARN] yamux: failed to send ping reply: session shutdown
2021/06/04 10:17:26 [ERR] agent: Coordinate update error: No cluster leader
2021/06/04 10:17:26 [ERR] memberlist: Failed fallback ping: write tcp 10.233.69.31:33976->10.233.74.2:8301: i/o timeout
2021/06/04 10:17:26 [INFO] memberlist: Suspect onap-consul-server-1 has failed, no acks received
2021/06/04 10:17:26 [INFO] consul: New leader elected: onap-consul-server-2
2021/06/04 10:17:27 [INFO] raft: Duplicate RequestVote for same term: 3
2021/06/04 10:17:30 [DEBUG] raft-net: 10.233.69.31:8300 accepted connection from: 10.233.77.160:34314
2021/06/04 10:31:35 [ERR] memberlist: Failed fallback ping: read tcp 10.233.69.31:46066->10.233.74.2:8301: i/o timeout
2021/06/04 10:31:35 [INFO] memberlist: Suspect onap-consul-server-1 has failed, no acks received
2021/06/04 10:31:38 [ERR] memberlist: Failed fallback ping: read tcp 10.233.69.31:46098->10.233.74.2:8301: i/o timeout
2021/06/04 10:31:38 [INFO] memberlist: Suspect onap-consul-server-1 has failed, no acks received
2021/06/04 10:31:39 [INFO] memberlist: Marking onap-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/06/04 10:31:39 [INFO] serf: EventMemberFailed: onap-consul-server-1 10.233.74.2
2021/06/04 10:31:39 [INFO] consul: Removing LAN server onap-consul-server-1 (Addr: tcp/10.233.74.2:8300) (DC: dc1)
2021/06/04 10:31:42 [ERR] memberlist: Failed fallback ping: read tcp 10.233.69.31:52324->10.233.74.2:8302: i/o timeout
2021/06/04 10:31:42 [INFO] memberlist: Suspect onap-consul-server-1.dc1 has failed, no acks received
2021/06/04 10:31:48 [WARN] memberlist: Refuting a suspect message (from: onap-consul-server-1)
2021/06/04 10:31:49 [INFO] serf: EventMemberJoin: onap-consul-server-1 10.233.74.2
2021/06/04 10:31:49 [INFO] consul: Adding LAN server onap-consul-server-1 (Addr: tcp/10.233.74.2:8300) (DC: dc1)
2021/06/04 10:31:54 [ERR] memberlist: Failed fallback ping: read tcp 10.233.69.31:46256->10.233.74.2:8301: i/o timeout
2021/06/04 10:31:54 [INFO] memberlist: Suspect onap-consul-server-1 has failed, no acks received
2021/06/04 10:31:56 [INFO] memberlist: Marking onap-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/06/04 10:31:56 [INFO] serf: EventMemberFailed: onap-consul-server-1 10.233.74.2
2021/06/04 10:31:56 [INFO] consul: Removing LAN server onap-consul-server-1 (Addr: tcp/10.233.74.2:8300) (DC: dc1)
2021/06/04 10:31:57 [ERR] memberlist: Failed fallback ping: read tcp 10.233.69.31:46276->10.233.74.2:8301: i/o timeout
2021/06/04 10:31:57 [INFO] memberlist: Suspect onap-consul-server-1 has failed, no acks received
2021/06/04 10:32:02 [ERR] memberlist: Failed fallback ping: read tcp 10.233.69.31:52526->10.233.74.2:8302: i/o timeout
2021/06/04 10:32:02 [INFO] memberlist: Suspect onap-consul-server-1.dc1 has failed, no acks received
2021/06/04 10:32:12 [ERR] memberlist: Failed fallback ping: read tcp 10.233.69.31:52624->10.233.74.2:8302: i/o timeout
2021/06/04 10:32:12 [INFO] memberlist: Suspect onap-consul-server-1.dc1 has failed, no acks received
2021/06/04 10:32:19 [INFO] serf: EventMemberJoin: onap-consul-server-1 10.233.74.2
2021/06/04 10:32:19 [INFO] consul: Adding LAN server onap-consul-server-1 (Addr: tcp/10.233.74.2:8300) (DC: dc1)
2021/06/04 10:32:22 [ERR] memberlist: Failed fallback ping: read tcp 10.233.69.31:46664->10.233.74.2:8301: i/o timeout
2021/06/04 10:32:22 [INFO] memberlist: Suspect onap-consul-server-1 has failed, no acks received
2021/06/04 10:32:24 [INFO] memberlist: Marking onap-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/06/04 10:32:24 [INFO] serf: EventMemberFailed: onap-consul-server-1 10.233.74.2
2021/06/04 10:32:24 [INFO] consul: Removing LAN server onap-consul-server-1 (Addr: tcp/10.233.74.2:8300) (DC: dc1)
2021/06/04 10:32:25 [INFO] serf: EventMemberJoin: onap-consul-server-1 10.233.74.2
2021/06/04 10:32:25 [INFO] consul: Adding LAN server onap-consul-server-1 (Addr: tcp/10.233.74.2:8300) (DC: dc1)
2021/06/04 10:32:25 [WARN] memberlist: Refuting a suspect message (from: onap-consul-server-1)
2021/06/04 10:33:12 [ERR] memberlist: Failed fallback ping: write tcp 10.233.69.31:47374->10.233.74.2:8301: i/o timeout
2021/06/04 10:33:12 [INFO] memberlist: Suspect onap-consul-server-1 has failed, no acks received
2021/06/04 10:33:22 [ERR] memberlist: Failed fallback ping: read tcp 10.233.69.31:47468->10.233.74.2:8301: i/o timeout
2021/06/04 10:33:22 [INFO] memberlist: Suspect onap-consul-server-1 has failed, no acks received
2021/06/04 10:33:26 [ERR] memberlist: Failed fallback ping: read tcp 10.233.69.31:47518->10.233.74.2:8301: i/o timeout
2021/06/04 10:33:26 [INFO] memberlist: Suspect onap-consul-server-1 has failed, no acks received
2021/06/04 10:33:26 [INFO] memberlist: Marking onap-consul-server-1 as failed, suspect timeout reached (2 peer confirmations)
2021/06/04 10:33:26 [INFO] serf: EventMemberFailed: onap-consul-server-1 10.233.74.2
2021/06/04 10:33:26 [INFO] consul: Removing LAN server onap-consul-server-1 (Addr: tcp/10.233.74.2:8300) (DC: dc1)
2021/06/04 10:33:27 [ERR] memberlist: Failed fallback ping: read tcp 10.233.69.31:53710->10.233.74.2:8302: i/o timeout
2021/06/04 10:33:27 [INFO] memberlist: Suspect onap-consul-server-1.dc1 has failed, no acks received
2021/06/04 10:33:42 [ERR] memberlist: Failed fallback ping: read tcp 10.233.69.31:53846->10.233.74.2:8302: i/o timeout
2021/06/04 10:33:42 [INFO] memberlist: Suspect onap-consul-server-1.dc1 has failed, no acks received
2021/06/04 10:33:52 [ERR] yamux: keepalive failed: i/o deadline reached
2021/06/04 10:33:52 [ERR] memberlist: Failed fallback ping: read tcp 10.233.69.31:53938->10.233.74.2:8302: i/o timeout
2021/06/04 10:33:52 [INFO] memberlist: Suspect onap-consul-server-1.dc1 has failed, no acks received
2021/06/04 10:33:57 [INFO] memberlist: Marking onap-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/06/04 10:33:57 [INFO] serf: EventMemberFailed: onap-consul-server-1.dc1 10.233.74.2
2021/06/04 10:33:57 [INFO] consul: Handled member-failed event for server "onap-consul-server-1.dc1" in area "wan"
2021/06/04 10:34:02 [ERR] memberlist: Failed fallback ping: read tcp 10.233.69.31:54052->10.233.74.2:8302: i/o timeout
2021/06/04 10:34:02 [INFO] memberlist: Suspect onap-consul-server-1.dc1 has failed, no acks received
2021/06/04 10:34:27 [INFO] serf: attempting reconnect to onap-consul-server-1.dc1 10.233.74.2:8302
2021/06/04 10:35:30 [INFO] serf: attempting reconnect to onap-consul-server-1 10.233.74.2:8301
2021/06/04 10:36:29 [INFO] serf: EventMemberJoin: onap-consul-server-1.dc1 10.233.74.2
2021/06/04 10:36:29 [INFO] serf: EventMemberJoin: onap-consul-server-1 10.233.74.2
2021/06/04 10:36:29 [INFO] consul: Adding LAN server onap-consul-server-1 (Addr: tcp/10.233.74.2:8300) (DC: dc1)
2021/06/04 10:36:29 [INFO] consul: Handled member-join event for server "onap-consul-server-1.dc1" in area "wan"
2021/06/04 10:36:33 [ERR] memberlist: Failed fallback ping: read tcp 10.233.69.31:49270->10.233.74.2:8301: i/o timeout
2021/06/04 10:36:33 [INFO] memberlist: Suspect onap-consul-server-1 has failed, no acks received
2021/06/04 10:36:37 [INFO] serf: EventMemberFailed: onap-consul-server-1 10.233.74.2
2021/06/04 10:36:37 [INFO] consul: Removing LAN server onap-consul-server-1 (Addr: tcp/10.233.74.2:8300) (DC: dc1)
2021/06/04 10:36:37 [ERR] memberlist: Failed fallback ping: read tcp 10.233.69.31:55496->10.233.74.2:8302: i/o timeout
2021/06/04 10:36:37 [INFO] memberlist: Suspect onap-consul-server-1.dc1 has failed, no acks received
2021/06/04 10:36:37 [ERR] memberlist: Failed fallback ping: read tcp 10.233.69.31:49306->10.233.74.2:8301: i/o timeout
2021/06/04 10:36:37 [INFO] memberlist: Suspect onap-consul-server-1 has failed, no acks received
2021/06/04 10:37:07 [INFO] memberlist: Marking onap-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations)
2021/06/04 10:37:07 [INFO] serf: EventMemberFailed: onap-consul-server-1.dc1 10.233.74.2
2021/06/04 10:37:07 [INFO] consul: Handled member-failed event for server "onap-consul-server-1.dc1" in area "wan"
2021/06/04 10:37:10 [INFO] serf: attempting reconnect to onap-consul-server-1 10.233.74.2:8301
2021/06/04 10:37:15 [INFO] memberlist: Suspect onap-consul-server-1.dc1 has failed, no acks received
2021/06/04 10:37:27 [INFO] serf: attempting reconnect to onap-consul-server-1.dc1 10.233.74.2:8302
2021/06/04 10:38:27 [INFO] serf: attempting reconnect to onap-consul-server-1.dc1 10.233.74.2:8302
2021/06/04 10:39:05 [INFO] serf: EventMemberJoin: onap-consul-server-1.dc1 10.233.74.2
2021/06/04 10:39:05 [INFO] consul: Handled member-join event for server "onap-consul-server-1.dc1" in area "wan"
2021/06/04 10:39:08 [INFO] serf: EventMemberJoin: onap-consul-server-1 10.233.74.2
2021/06/04 10:39:08 [INFO] consul: Adding LAN server onap-consul-server-1 (Addr: tcp/10.233.74.2:8300) (DC: dc1)
2021/06/04 11:08:58 [ERR] yamux: keepalive failed: session shutdown