2021-06-07 14:00:07,189|None|139923588508448|INFO|service|conductor.reservation.service: [-] plan id: healthcheck [-] Reservation starts, changing the template status from solved to reserving, atomic update response from MUSIC SUCCESS 2021-06-07 14:00:07,190|None|139923588508448|INFO|service|conductor.reservation.service: [-] plan id: healthcheck [-] Plan healthcheck with request id foo is reserving by machine dev-oof-has-reservation-55b5d85b78-jnd57. Tried to reserve it for 1 times. 2021-06-07 14:00:07,190|None|139923588508448|ERROR|service|conductor.reservation.service: [-] plan id: healthcheck [-] no conductor_solver in translation for Plan healthcheck 2021-06-07 14:00:07,191|None|139923588508448|INFO|service|conductor.reservation.service: [-] plan id: healthcheck [-] Plan healthcheck with request id foo Reservation complete 2021-06-07 14:00:08,590|None|139923588508448|INFO|service|conductor.reservation.service: [-] plan id: healthcheck [-] Reservation is complete, changing the template status from reserving to done, atomic update response from MUSIC SUCCESS 2021-06-07 14:00:59,694|None|139923588508448|ERROR|_utils|cotyledon._utils: [-] plan id: healthcheck [-] Unhandled exception Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623074459.693664055","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 265, in run reserving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.RESERVING) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read return self.get_values_prefix(key_prefix, pk_name, pk_value) File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix for kv in self.get_client().get_prefix(key_prefix)} File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 14:00:59,702||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 152 exited with status 2 2021-06-07 14:03:00,832||139923588508448|ERROR|_utils|cotyledon._utils: [-] Unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler return f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 257, in get_response return self.kvstub.Range( File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623074580.832078169","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 265, in run reserving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.RESERVING) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read schema = self.get_value(f'{keyspace}/{table}') File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value raw_value = self.get_raw_value(key) File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value return self.get_client().get(key)[0] File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 283, in get range_response = self.get_response(key, **kwargs) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler _translate_exception(exc) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception raise exception etcd3.exceptions.ConnectionFailedError: etcd connection failed 2021-06-07 14:03:00,840||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 490 exited with status 2 Process Process-3: Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623074594.832494117","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait sw = cls(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__ self.service = config.service(worker_id, *args, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__ self._init(conf, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init self.data_service = self.setup_rpc(conf, "data") File "/opt/has/conductor/conductor/reservation/service.py", line 178, in setup_rpc transport = messaging.get_transport(conf=conf) File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport transport.keyspace_create(keyspace=keyspace) File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode()) File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 14:03:14,842||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 768 exited with status 1 2021-06-07 14:04:11,713||139923588508448|ERROR|_utils|cotyledon._utils: [-] Unhandled exception Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623074651.712027019","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read return self.get_values_prefix(key_prefix, pk_name, pk_value) File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix for kv in self.get_client().get_prefix(key_prefix)} File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 14:04:11,722||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 796 exited with status 2 Process Process-5: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler return f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 834, in transaction txn_response = self.kvstub.Txn( File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623074663.991120835","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait sw = cls(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__ self.service = config.service(worker_id, *args, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__ self._init(conf, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init self.data_service = self.setup_rpc(conf, "data") File "/opt/has/conductor/conductor/reservation/service.py", line 178, in setup_rpc transport = messaging.get_transport(conf=conf) File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport transport.keyspace_create(keyspace=keyspace) File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode()) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler return f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 444, in put_if_not_exists status, _ = self.transaction( File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler _translate_exception(exc) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception raise exception etcd3.exceptions.ConnectionFailedError: etcd connection failed 2021-06-07 14:04:24,004||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 892 exited with status 1 Process Process-6: Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623074675.429381117","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait sw = cls(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__ self.service = config.service(worker_id, *args, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__ self._init(conf, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init self.data_service = self.setup_rpc(conf, "data") File "/opt/has/conductor/conductor/reservation/service.py", line 178, in setup_rpc transport = messaging.get_transport(conf=conf) File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport transport.keyspace_create(keyspace=keyspace) File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode()) File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 14:04:35,437||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 919 exited with status 1 Process Process-7: Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623074687.261453890","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait sw = cls(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__ self.service = config.service(worker_id, *args, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__ self._init(conf, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init self.data_service = self.setup_rpc(conf, "data") File "/opt/has/conductor/conductor/reservation/service.py", line 178, in setup_rpc transport = messaging.get_transport(conf=conf) File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport transport.keyspace_create(keyspace=keyspace) File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode()) File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 14:04:47,268||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 936 exited with status 1 2021-06-07 14:08:44,041||139923588508448|ERROR|_utils|cotyledon._utils: [-] Unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler return f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 257, in get_response return self.kvstub.Range( File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623074924.040057825","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read schema = self.get_value(f'{keyspace}/{table}') File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value raw_value = self.get_raw_value(key) File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value return self.get_client().get(key)[0] File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 283, in get range_response = self.get_response(key, **kwargs) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler _translate_exception(exc) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception raise exception etcd3.exceptions.ConnectionFailedError: etcd connection failed 2021-06-07 14:08:44,049||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 953 exited with status 2 2021-06-07 14:12:19,091||139923588508448|ERROR|_utils|cotyledon._utils: [-] Unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler return f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 302, in get_prefix_response return self.kvstub.Range( File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623075139.090557634","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read return self.get_values_prefix(key_prefix, pk_name, pk_value) File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix for kv in self.get_client().get_prefix(key_prefix)} File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 318, in get_prefix range_response = self.get_prefix_response(key_prefix, **kwargs) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler _translate_exception(exc) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception raise exception etcd3.exceptions.ConnectionFailedError: etcd connection failed 2021-06-07 14:12:19,099||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 1793 exited with status 2 2021-06-07 14:15:12,484||139923588508448|ERROR|_utils|cotyledon._utils: [-] Unhandled exception Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623075312.481477293","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read schema = self.get_value(f'{keyspace}/{table}') File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value raw_value = self.get_raw_value(key) File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value return self.get_client().get(key)[0] File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 14:15:12,493||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 2527 exited with status 2 2021-06-07 14:18:03,499||139923588508448|ERROR|_utils|cotyledon._utils: [-] Unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler return f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 302, in get_prefix_response return self.kvstub.Range( File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623075483.498147512","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read return self.get_values_prefix(key_prefix, pk_name, pk_value) File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix for kv in self.get_client().get_prefix(key_prefix)} File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 318, in get_prefix range_response = self.get_prefix_response(key_prefix, **kwargs) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler _translate_exception(exc) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception raise exception etcd3.exceptions.ConnectionFailedError: etcd connection failed 2021-06-07 14:18:03,527||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 3047 exited with status 2 Process Process-12: Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623075497.498918327","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait sw = cls(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__ self.service = config.service(worker_id, *args, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__ self._init(conf, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init self.data_service = self.setup_rpc(conf, "data") File "/opt/has/conductor/conductor/reservation/service.py", line 178, in setup_rpc transport = messaging.get_transport(conf=conf) File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport transport.keyspace_create(keyspace=keyspace) File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode()) File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 14:18:17,508||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 3443 exited with status 1 Process Process-13: Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623075504.649049590","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait sw = cls(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__ self.service = config.service(worker_id, *args, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__ self._init(conf, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init self.data_service = self.setup_rpc(conf, "data") File "/opt/has/conductor/conductor/reservation/service.py", line 178, in setup_rpc transport = messaging.get_transport(conf=conf) File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport transport.keyspace_create(keyspace=keyspace) File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode()) File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 14:18:24,662||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 3465 exited with status 1 2021-06-07 14:19:59,660||139923588508448|ERROR|_utils|cotyledon._utils: [-] Unhandled exception Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623075599.659337200","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 265, in run reserving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.RESERVING) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read return self.get_values_prefix(key_prefix, pk_name, pk_value) File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix for kv in self.get_client().get_prefix(key_prefix)} File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 14:19:59,668||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 3482 exited with status 2 2021-06-07 14:25:13,972||139923588508448|ERROR|_utils|cotyledon._utils: [-] Unhandled exception Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623075913.970381015","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 265, in run reserving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.RESERVING) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read schema = self.get_value(f'{keyspace}/{table}') File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value raw_value = self.get_raw_value(key) File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value return self.get_client().get(key)[0] File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 14:25:13,980||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 3650 exited with status 2 Process Process-16: Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623075921.824883255","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait sw = cls(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__ self.service = config.service(worker_id, *args, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__ self._init(conf, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init self.data_service = self.setup_rpc(conf, "data") File "/opt/has/conductor/conductor/reservation/service.py", line 178, in setup_rpc transport = messaging.get_transport(conf=conf) File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport transport.keyspace_create(keyspace=keyspace) File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode()) File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 14:25:21,837||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 4795 exited with status 1 Process Process-17: Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623075935.825688549","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait sw = cls(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__ self.service = config.service(worker_id, *args, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__ self._init(conf, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init self.data_service = self.setup_rpc(conf, "data") File "/opt/has/conductor/conductor/reservation/service.py", line 178, in setup_rpc transport = messaging.get_transport(conf=conf) File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport transport.keyspace_create(keyspace=keyspace) File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode()) File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 14:25:35,838||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 4801 exited with status 1 Process Process-18: Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623075942.857340281","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait sw = cls(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__ self.service = config.service(worker_id, *args, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__ self._init(conf, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init self.data_service = self.setup_rpc(conf, "data") File "/opt/has/conductor/conductor/reservation/service.py", line 178, in setup_rpc transport = messaging.get_transport(conf=conf) File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport transport.keyspace_create(keyspace=keyspace) File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode()) File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 14:25:42,863||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 4831 exited with status 1 2021-06-07 14:27:36,656||139923588508448|ERROR|_utils|cotyledon._utils: [-] Unhandled exception Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623076056.655079658","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read schema = self.get_value(f'{keyspace}/{table}') File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value raw_value = self.get_raw_value(key) File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value return self.get_client().get(key)[0] File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 14:27:36,663||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 4842 exited with status 2 2021-06-07 14:28:47,098||139923588508448|ERROR|_utils|cotyledon._utils: [-] Unhandled exception Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623076127.097608083","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read return self.get_values_prefix(key_prefix, pk_name, pk_value) File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix for kv in self.get_client().get_prefix(key_prefix)} File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 14:28:47,106||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 5222 exited with status 2 2021-06-07 14:29:48,013||139923588508448|ERROR|_utils|cotyledon._utils: [-] Unhandled exception Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623076188.011047566","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 265, in run reserving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.RESERVING) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read return self.get_values_prefix(key_prefix, pk_name, pk_value) File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix for kv in self.get_client().get_prefix(key_prefix)} File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 14:29:48,022||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 5442 exited with status 2 Process Process-22: Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623076195.047320370","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait sw = cls(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__ self.service = config.service(worker_id, *args, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__ self._init(conf, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init self.data_service = self.setup_rpc(conf, "data") File "/opt/has/conductor/conductor/reservation/service.py", line 178, in setup_rpc transport = messaging.get_transport(conf=conf) File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport transport.keyspace_create(keyspace=keyspace) File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode()) File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 14:29:55,054||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 5583 exited with status 1 2021-06-07 14:30:39,348||139923588508448|ERROR|_utils|cotyledon._utils: [-] Unhandled exception Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623076239.347286639","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read return self.get_values_prefix(key_prefix, pk_name, pk_value) File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix for kv in self.get_client().get_prefix(key_prefix)} File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 14:30:39,354||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 5600 exited with status 2 Process Process-24: Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623076252.572638746","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait sw = cls(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__ self.service = config.service(worker_id, *args, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__ self._init(conf, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init self.data_service = self.setup_rpc(conf, "data") File "/opt/has/conductor/conductor/reservation/service.py", line 178, in setup_rpc transport = messaging.get_transport(conf=conf) File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport transport.keyspace_create(keyspace=keyspace) File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode()) File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 14:30:52,583||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 5686 exited with status 1 Process Process-25: Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623076262.379806892","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait sw = cls(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__ self.service = config.service(worker_id, *args, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__ self._init(conf, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init self.data_service = self.setup_rpc(conf, "data") File "/opt/has/conductor/conductor/reservation/service.py", line 178, in setup_rpc transport = messaging.get_transport(conf=conf) File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport transport.keyspace_create(keyspace=keyspace) File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode()) File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 14:31:02,392||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 5702 exited with status 1 2021-06-07 14:35:55,866||139923588508448|ERROR|_utils|cotyledon._utils: [-] Unhandled exception Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623076555.865780899","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read schema = self.get_value(f'{keyspace}/{table}') File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value raw_value = self.get_raw_value(key) File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value return self.get_client().get(key)[0] File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 14:35:55,874||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 5718 exited with status 2 2021-06-07 14:41:44,670||139923588508448|ERROR|_utils|cotyledon._utils: [-] Unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler return f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 257, in get_response return self.kvstub.Range( File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623076904.669381665","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 265, in run reserving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.RESERVING) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read schema = self.get_value(f'{keyspace}/{table}') File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value raw_value = self.get_raw_value(key) File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value return self.get_client().get(key)[0] File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 283, in get range_response = self.get_response(key, **kwargs) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler _translate_exception(exc) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception raise exception etcd3.exceptions.ConnectionFailedError: etcd connection failed 2021-06-07 14:41:44,678||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 6845 exited with status 2 Process Process-28: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler return f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 834, in transaction txn_response = self.kvstub.Txn( File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623076915.177365369","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait sw = cls(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__ self.service = config.service(worker_id, *args, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__ self._init(conf, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init self.data_service = self.setup_rpc(conf, "data") File "/opt/has/conductor/conductor/reservation/service.py", line 178, in setup_rpc transport = messaging.get_transport(conf=conf) File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport transport.keyspace_create(keyspace=keyspace) File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode()) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler return f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 444, in put_if_not_exists status, _ = self.transaction( File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler _translate_exception(exc) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception raise exception etcd3.exceptions.ConnectionFailedError: etcd connection failed 2021-06-07 14:41:55,187||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 7939 exited with status 1 2021-06-07 14:42:17,683||139923588508448|ERROR|_utils|cotyledon._utils: [-] Unhandled exception Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623076937.681915573","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read return self.get_values_prefix(key_prefix, pk_name, pk_value) File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix for kv in self.get_client().get_prefix(key_prefix)} File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 14:42:17,693||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 7955 exited with status 2 2021-06-07 14:59:18,003||139923588508448|ERROR|_utils|cotyledon._utils: [-] Unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler return f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 257, in get_response return self.kvstub.Range( File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623077958.003002791","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 265, in run reserving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.RESERVING) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read schema = self.get_value(f'{keyspace}/{table}') File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value raw_value = self.get_raw_value(key) File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value return self.get_client().get(key)[0] File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 283, in get range_response = self.get_response(key, **kwargs) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler _translate_exception(exc) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception raise exception etcd3.exceptions.ConnectionFailedError: etcd connection failed 2021-06-07 14:59:18,012||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 8004 exited with status 2 2021-06-07 15:03:48,965||139923588508448|ERROR|_utils|cotyledon._utils: [-] Unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler return f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 302, in get_prefix_response return self.kvstub.Range( File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623078228.964009609","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 265, in run reserving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.RESERVING) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read return self.get_values_prefix(key_prefix, pk_name, pk_value) File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix for kv in self.get_client().get_prefix(key_prefix)} File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 318, in get_prefix range_response = self.get_prefix_response(key_prefix, **kwargs) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler _translate_exception(exc) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception raise exception etcd3.exceptions.ConnectionFailedError: etcd connection failed 2021-06-07 15:03:48,973||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 11263 exited with status 2 Process Process-32: Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623078238.916895070","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait sw = cls(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__ self.service = config.service(worker_id, *args, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__ self._init(conf, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init self.data_service = self.setup_rpc(conf, "data") File "/opt/has/conductor/conductor/reservation/service.py", line 178, in setup_rpc transport = messaging.get_transport(conf=conf) File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport transport.keyspace_create(keyspace=keyspace) File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode()) File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 15:03:58,923||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 12077 exited with status 1 2021-06-07 15:12:14,382||139923588508448|ERROR|_utils|cotyledon._utils: [-] Unhandled exception Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623078734.381391241","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read return self.get_values_prefix(key_prefix, pk_name, pk_value) File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix for kv in self.get_client().get_prefix(key_prefix)} File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 15:12:14,390||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 12093 exited with status 2 Process Process-34: Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623078745.986364200","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait sw = cls(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__ self.service = config.service(worker_id, *args, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__ self._init(conf, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init self.data_service = self.setup_rpc(conf, "data") File "/opt/has/conductor/conductor/reservation/service.py", line 178, in setup_rpc transport = messaging.get_transport(conf=conf) File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport transport.keyspace_create(keyspace=keyspace) File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode()) File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 15:12:25,996||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 13446 exited with status 1 Process Process-35: Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623078755.653397792","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait sw = cls(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__ self.service = config.service(worker_id, *args, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__ self._init(conf, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init self.data_service = self.setup_rpc(conf, "data") File "/opt/has/conductor/conductor/reservation/service.py", line 178, in setup_rpc transport = messaging.get_transport(conf=conf) File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport transport.keyspace_create(keyspace=keyspace) File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode()) File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 15:12:35,665||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 13462 exited with status 1 2021-06-07 15:18:45,311||139923588508448|ERROR|_utils|cotyledon._utils: [-] Unhandled exception Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623079125.310511043","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read schema = self.get_value(f'{keyspace}/{table}') File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value raw_value = self.get_raw_value(key) File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value return self.get_client().get(key)[0] File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 15:18:45,318||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 13479 exited with status 2 Process Process-37: Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623079132.340634813","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait sw = cls(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__ self.service = config.service(worker_id, *args, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__ self._init(conf, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init self.data_service = self.setup_rpc(conf, "data") File "/opt/has/conductor/conductor/reservation/service.py", line 178, in setup_rpc transport = messaging.get_transport(conf=conf) File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport transport.keyspace_create(keyspace=keyspace) File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode()) File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 15:18:52,350||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 14543 exited with status 1 2021-06-07 15:23:07,365||139923588508448|ERROR|_utils|cotyledon._utils: [-] Unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler return f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 257, in get_response return self.kvstub.Range( File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623079387.363825491","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read schema = self.get_value(f'{keyspace}/{table}') File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value raw_value = self.get_raw_value(key) File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value return self.get_client().get(key)[0] File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 283, in get range_response = self.get_response(key, **kwargs) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler _translate_exception(exc) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception raise exception etcd3.exceptions.ConnectionFailedError: etcd connection failed 2021-06-07 15:23:07,374||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 14549 exited with status 2 2021-06-07 15:24:44,920||139923588508448|ERROR|_utils|cotyledon._utils: [-] Unhandled exception Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler return f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 302, in get_prefix_response return self.kvstub.Range( File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623079484.917905043","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 265, in run reserving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.RESERVING) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read return self.get_values_prefix(key_prefix, pk_name, pk_value) File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix for kv in self.get_client().get_prefix(key_prefix)} File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 318, in get_prefix range_response = self.get_prefix_response(key_prefix, **kwargs) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler _translate_exception(exc) File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception raise exception etcd3.exceptions.ConnectionFailedError: etcd connection failed 2021-06-07 15:24:44,929||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 15335 exited with status 2 Process Process-40: Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623079494.551830845","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait sw = cls(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__ self.service = config.service(worker_id, *args, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__ self._init(conf, **kwargs) File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init self.data_service = self.setup_rpc(conf, "data") File "/opt/has/conductor/conductor/reservation/service.py", line 178, in setup_rpc transport = messaging.get_transport(conf=conf) File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport transport.keyspace_create(keyspace=keyspace) File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode()) File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 15:24:54,563||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 15630 exited with status 1 2021-06-07 15:33:06,286||139923588508448|ERROR|_utils|cotyledon._utils: [-] Unhandled exception Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623079986.285600786","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read return self.get_values_prefix(key_prefix, pk_name, pk_value) File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix for kv in self.get_client().get_prefix(key_prefix)} File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 15:33:06,294||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 15648 exited with status 2 2021-06-07 15:33:26,519||139923588508448|ERROR|_utils|cotyledon._utils: [-] Unhandled exception Traceback (most recent call last): File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client return etcd3.client(host=self.host, port=self.port, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client return Etcd3Client(host=host, File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__ resp = self.auth_stub.Authenticate(auth_request, self.timeout) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "etcdserver: request timed out" debug_error_string = "{"created":"@1623080006.514453183","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception yield File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run self.run() File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED) File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col rows = db_backend.DB_API.row_read( File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read schema = self.get_value(f'{keyspace}/{table}') File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value raw_value = self.get_raw_value(key) File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value return self.get_client().get(key)[0] File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code())) conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE 2021-06-07 15:33:26,524||139923590630176|INFO|_service_manager|cotyledon._service_manager: [-] Child 17061 exited with status 2