+ CONSUL_BIND= + [ -n ] + CONSUL_CLIENT= + [ -n ] + CONSUL_DATA_DIR=/consul/data + CONSUL_CONFIG_DIR=/consul/config + [ -n ] + [ a = - ] + [ agent = agent ] + shift + set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui + [ consul = consul ] + stat -c %u /consul/data + id -u consul + [ 100 != 100 ] + stat -c %u /consul/config + id -u consul + [ 100 != 100 ] + [ ! -z ] + exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui bootstrap_expect > 0: expecting 3 servers ==> Starting Consul agent... ==> Consul agent running! Version: 'v1.0.6' Node ID: '59c594ab-4dd4-45c8-f46f-385adf64f752' Node name: 'dev-consul-server-1' Datacenter: 'dc1' (Segment: '') Server: true (Bootstrap: false) Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600) Cluster Addr: 10.242.149.73 (LAN: 8301, WAN: 8302) Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false ==> Log data will now stream in as it occurs: 2021/04/12 22:40:52 [INFO] raft: Initial configuration (index=0): [] 2021/04/12 22:40:52 [INFO] raft: Node at 10.242.149.73:8300 [Follower] entering Follower state (Leader: "") 2021/04/12 22:40:52 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.149.73 2021/04/12 22:40:52 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.149.73 2021/04/12 22:40:52 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan" 2021/04/12 22:40:52 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.149.73:8300) (DC: dc1) 2021/04/12 22:40:52 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp) 2021/04/12 22:40:52 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp) 2021/04/12 22:40:52 [INFO] agent: Started HTTP server on [::]:8500 (tcp) 2021/04/12 22:40:52 [INFO] agent: started state syncer 2021/04/12 22:40:52 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer 2021/04/12 22:40:52 [INFO] agent: Joining LAN cluster... 2021/04/12 22:40:52 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server] 2021/04/12 22:40:52 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.240.138 2021/04/12 22:40:52 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.240.138:8300) (DC: dc1) 2021/04/12 22:40:52 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.240.138 2021/04/12 22:40:52 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan" 2021/04/12 22:40:52 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host 2021/04/12 22:40:52 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host 2021/04/12 22:40:52 [INFO] agent: (LAN) joined: 1 Err: 2021/04/12 22:40:52 [INFO] agent: Join LAN completed. Synced with 1 initial agents 2021/04/12 22:40:59 [ERR] agent: failed to sync remote state: No cluster leader 2021/04/12 22:41:00 [WARN] raft: no known peers, aborting election 2021/04/12 22:41:06 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.53.8 2021/04/12 22:41:06 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan" 2021/04/12 22:41:06 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.53.8 2021/04/12 22:41:06 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.53.8:8300) (DC: dc1) 2021/04/12 22:41:06 [INFO] consul: Existing Raft peers reported by dev-consul-server-0, disabling bootstrap mode 2021/04/12 22:41:06 [DEBUG] raft-net: 10.242.149.73:8300 accepted connection from: 10.242.240.138:47000 2021/04/12 22:41:06 [DEBUG] raft-net: 10.242.149.73:8300 accepted connection from: 10.242.240.138:47004 2021/04/12 22:41:06 [WARN] raft: Failed to get previous log: 1 log not found (last: 0) 2021/04/12 22:41:07 [INFO] consul: New leader elected: dev-consul-server-0 2021/04/12 22:41:09 [INFO] agent: Synced node info ==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=38b83eb1-7915-c59e-fdb8-5d789dee6cfb&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 2021/04/12 22:41:17 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-qtrtq 10.242.240.139 2021/04/12 23:05:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.149.73:44616->10.242.53.8:8301: i/o timeout 2021/04/12 23:05:05 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/04/12 23:05:07 [ERR] memberlist: Failed fallback ping: read tcp 10.242.149.73:41916->10.242.53.8:8302: i/o timeout 2021/04/12 23:05:07 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received 2021/04/12 23:05:07 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.53.8 2021/04/12 23:05:07 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.53.8:8300) (DC: dc1) 2021/04/12 23:05:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.149.73:44664->10.242.53.8:8301: i/o timeout 2021/04/12 23:05:08 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/04/12 23:05:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.149.73:42048->10.242.53.8:8302: i/o timeout 2021/04/12 23:05:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received 2021/04/12 23:05:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.149.73:42188->10.242.53.8:8302: i/o timeout 2021/04/12 23:05:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received 2021/04/12 23:05:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.149.73:42328->10.242.53.8:8302: i/o timeout 2021/04/12 23:05:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received 2021/04/12 23:05:37 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/04/12 23:05:37 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.53.8 2021/04/12 23:05:37 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan" 2021/04/12 23:05:52 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.53.8:8301 2021/04/12 23:05:59 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/12 23:06:22 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.53.8:8302 2021/04/12 23:07:22 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.53.8:8302 2021/04/12 23:08:52 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.53.8:8302 2021/04/12 23:09:52 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.53.8:8302 2021/04/12 23:10:52 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.53.8:8302 2021/04/12 23:11:02 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.53.8:8301 2021/04/12 23:11:47 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/12 23:11:52 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.53.8:8302 2021/04/12 23:12:42 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.53.8:8301 2021/04/12 23:13:13 [ERR] memberlist: failed to receive: read tcp 10.242.149.73:8302->10.242.53.8:45520: i/o timeout from=10.242.53.8:45520 2021/04/12 23:13:13 [ERR] memberlist: Failed to send error: write tcp 10.242.149.73:8302->10.242.53.8:45520: i/o timeout from=10.242.53.8:45520 2021/04/12 23:13:52 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.53.8:8301 2021/04/12 23:14:32 [ERR] yamux: keepalive failed: i/o deadline reached 2021/04/12 23:14:52 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.53.8:8302 2021/04/12 23:15:02 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.53.8:8301 2021/04/12 23:15:10 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.53.8 2021/04/12 23:15:10 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan" 2021/04/12 23:15:17 [ERR] memberlist: Failed fallback ping: read tcp 10.242.149.73:51040->10.242.53.8:8302: i/o timeout 2021/04/12 23:15:17 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received 2021/04/12 23:15:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.149.73:51206->10.242.53.8:8302: i/o timeout 2021/04/12 23:15:27 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received 2021/04/12 23:15:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.149.73:51358->10.242.53.8:8302: i/o timeout 2021/04/12 23:15:37 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received 2021/04/12 23:15:47 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/04/12 23:15:47 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.53.8 2021/04/12 23:15:47 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan" 2021/04/12 23:15:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.149.73:51608->10.242.53.8:8302: i/o timeout 2021/04/12 23:15:52 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received 2021/04/12 23:15:52 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.53.8:8302 2021/04/12 23:15:56 [DEBUG] raft-net: 10.242.149.73:8300 accepted connection from: 10.242.53.8:35786 2021/04/12 23:15:56 [WARN] raft: Rejecting vote request from 10.242.53.8:8300 since we have a leader: 10.242.240.138:8300 2021/04/12 23:16:16 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.53.8 2021/04/12 23:16:16 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan" 2021/04/12 23:16:16 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.53.8 2021/04/12 23:16:16 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.53.8:8300) (DC: dc1) 2021/04/12 23:16:17 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2) 2021/04/12 23:16:18 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2.dc1) 2021/04/12 23:16:22 [ERR] memberlist: Failed fallback ping: read tcp 10.242.149.73:52092->10.242.53.8:8302: i/o timeout 2021/04/12 23:16:22 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received 2021/04/12 23:16:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.149.73:54852->10.242.53.8:8301: i/o timeout 2021/04/12 23:16:24 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/04/12 23:16:25 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.53.8 2021/04/12 23:16:25 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.53.8:8300) (DC: dc1) 2021/04/12 23:16:27 [ERR] memberlist: Failed fallback ping: read tcp 10.242.149.73:54860->10.242.53.8:8301: i/o timeout 2021/04/12 23:16:27 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/04/12 23:16:28 [ERR] memberlist: Push/Pull with dev-consul-server-2 failed: read tcp 10.242.149.73:54782->10.242.53.8:8301: i/o timeout 2021/04/12 23:16:29 [WARN] raft: Rejecting vote request from 10.242.53.8:8300 since we have a leader: 10.242.240.138:8300 2021/04/12 23:16:35 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.53.8 2021/04/12 23:16:35 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.53.8:8300) (DC: dc1) 2021/04/12 23:16:37 [WARN] memberlist: Was able to connect to dev-consul-server-2.dc1 but other probes failed, network may be misconfigured 2021/04/12 23:16:37 [INFO] consul: New leader elected: dev-consul-server-0 2021/04/12 23:16:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.149.73:55084->10.242.53.8:8301: i/o timeout 2021/04/12 23:16:38 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/04/12 23:16:38 [DEBUG] raft-net: 10.242.149.73:8300 accepted connection from: 10.242.240.138:53988 2021/04/12 23:16:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.149.73:55132->10.242.53.8:8301: i/o timeout 2021/04/12 23:16:41 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/04/12 23:16:42 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations) 2021/04/12 23:16:42 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.53.8 2021/04/12 23:16:42 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.53.8:8300) (DC: dc1) 2021/04/12 23:16:42 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.53.8:8301 2021/04/12 23:16:43 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-2) 2021/04/12 23:16:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.149.73:55152->10.242.53.8:8301: i/o timeout 2021/04/12 23:16:44 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/04/12 23:16:45 [WARN] raft: Rejecting vote request from 10.242.53.8:8300 since we have a leader: 10.242.240.138:8300 2021/04/12 23:16:45 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.53.8 2021/04/12 23:16:45 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.53.8:8300) (DC: dc1) 2021/04/12 23:16:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.149.73:55242->10.242.53.8:8301: i/o timeout 2021/04/12 23:16:48 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/04/12 23:16:52 [ERR] memberlist: Failed fallback ping: read tcp 10.242.149.73:55312->10.242.53.8:8301: i/o timeout 2021/04/12 23:16:52 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/04/12 23:16:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.149.73:55342->10.242.53.8:8301: i/o timeout 2021/04/12 23:16:55 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/04/12 23:16:56 [INFO] memberlist: Marking dev-consul-server-2 as failed, suspect timeout reached (2 peer confirmations) 2021/04/12 23:16:56 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.53.8 2021/04/12 23:16:56 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.53.8:8300) (DC: dc1) 2021/04/12 23:16:56 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.53.8 2021/04/12 23:16:56 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.53.8:8300) (DC: dc1) 2021/04/12 23:17:11 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/04/12 23:17:12 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received 2021/04/12 23:17:12 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.53.8 2021/04/12 23:17:12 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.53.8:8300) (DC: dc1) 2021/04/12 23:17:22 [INFO] memberlist: Suspect dev-consul-server-2 has failed, no acks received 2021/04/12 23:17:33 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.53.8:8302 Theirs: 10.242.53.46:8302 2021/04/12 23:17:33 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.53.8:8302 and 10.242.53.46:8302 are claiming 2021/04/12 23:17:33 [ERR] memberlist: Conflicting address for dev-consul-server-2. Mine: 10.242.53.8:8301 Theirs: 10.242.53.46:8301 2021/04/12 23:17:33 [WARN] serf: Name conflict for 'dev-consul-server-2' both 10.242.53.8:8301 and 10.242.53.46:8301 are claiming 2021/04/12 23:17:33 [ERR] memberlist: Conflicting address for dev-consul-server-2. Mine: 10.242.53.8:8301 Theirs: 10.242.53.46:8301 2021/04/12 23:17:33 [WARN] serf: Name conflict for 'dev-consul-server-2' both 10.242.53.8:8301 and 10.242.53.46:8301 are claiming 2021/04/12 23:17:33 [ERR] memberlist: Conflicting address for dev-consul-server-2. Mine: 10.242.53.8:8301 Theirs: 10.242.53.46:8301 2021/04/12 23:17:33 [WARN] serf: Name conflict for 'dev-consul-server-2' both 10.242.53.8:8301 and 10.242.53.46:8301 are claiming 2021/04/12 23:17:34 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.53.8:8302 Theirs: 10.242.53.46:8302 2021/04/12 23:17:34 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.53.8:8302 and 10.242.53.46:8302 are claiming 2021/04/12 23:17:34 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.53.8:8302 Theirs: 10.242.53.46:8302 2021/04/12 23:17:34 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.53.8:8302 and 10.242.53.46:8302 are claiming 2021/04/12 23:17:34 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.53.8:8302 Theirs: 10.242.53.46:8302 2021/04/12 23:17:34 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.53.8:8302 and 10.242.53.46:8302 are claiming 2021/04/12 23:17:34 [ERR] memberlist: Conflicting address for dev-consul-server-2.dc1. Mine: 10.242.53.8:8302 Theirs: 10.242.53.46:8302 2021/04/12 23:17:34 [WARN] serf: Name conflict for 'dev-consul-server-2.dc1' both 10.242.53.8:8302 and 10.242.53.46:8302 are claiming 2021/04/12 23:17:42 [INFO] memberlist: Marking dev-consul-server-2.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/04/12 23:17:42 [INFO] serf: EventMemberFailed: dev-consul-server-2.dc1 10.242.53.8 2021/04/12 23:17:42 [INFO] consul: Handled member-failed event for server "dev-consul-server-2.dc1" in area "wan" 2021/04/12 23:17:44 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.53.8:8301 2021/04/12 23:17:50 [INFO] memberlist: Suspect dev-consul-server-2.dc1 has failed, no acks received 2021/04/12 23:18:16 [INFO] serf: attempting reconnect to dev-consul-server-2.dc1 10.242.53.8:8302 2021/04/12 23:18:24 [INFO] serf: attempting reconnect to dev-consul-server-2 10.242.53.8:8301 2021/04/12 23:18:27 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.53.46 2021/04/12 23:18:27 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.53.46:8300) (DC: dc1) 2021/04/12 23:19:39 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.53.46 2021/04/12 23:19:39 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan"