2021-06-07 13:57:58,971||140579341892936|INFO|prometheus_metrics|conductor.common.prometheus_metrics: [-] Prometheus metrics endpoint started at 8001
2021-06-07 13:58:16,768||140579273841440|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623074296.768106596","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/solver/service.py", line 302, in run
    translated_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TRANSLATED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 13:58:16,775||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 165 exited with status 2
2021-06-07 13:58:45,915||140579273841440|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 257, in get_response
    return self.kvstub.Range(
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623074325.914369729","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/solver/service.py", line 302, in run
    translated_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TRANSLATED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 283, in get
    range_response = self.get_response(key, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
    _translate_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
    raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-06-07 13:58:45,922||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 204 exited with status 2
2021-06-07 14:00:59,615||140579273841440|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623074459.614983708","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/solver/service.py", line 303, in run
    solving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVING)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:00:59,623||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 250 exited with status 2
2021-06-07 14:03:07,832||140579273841440|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623074587.831902799","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/solver/service.py", line 302, in run
    translated_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TRANSLATED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:03:07,839||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 599 exited with status 2
Process Process-5:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623074594.857351736","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 203, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 219, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/solver/service.py", line 271, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:03:14,867||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 869 exited with status 1
2021-06-07 14:04:11,786||140579273841440|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623074651.785848210","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/solver/service.py", line 302, in run
    translated_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TRANSLATED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:04:11,793||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 880 exited with status 2
Process Process-7:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 834, in transaction
    txn_response = self.kvstub.Txn(
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623074663.987193915","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 203, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 219, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/solver/service.py", line 271, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 444, in put_if_not_exists
    status, _ = self.transaction(
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
    _translate_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
    raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-06-07 14:04:23,994||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 981 exited with status 1
Process Process-8:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623074673.192512335","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 203, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 219, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/solver/service.py", line 271, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:04:33,202||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 1004 exited with status 1
Process Process-9:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623074680.341346273","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 203, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 219, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/solver/service.py", line 271, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:04:40,348||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 1015 exited with status 1
2021-06-07 14:08:37,891||140579273841440|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 257, in get_response
    return self.kvstub.Range(
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623074917.890348188","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/solver/service.py", line 302, in run
    translated_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TRANSLATED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 283, in get
    range_response = self.get_response(key, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
    _translate_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
    raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-06-07 14:08:37,899||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 1031 exited with status 2
Process Process-11:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623074924.919237418","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 203, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 219, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/solver/service.py", line 271, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:08:44,929||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 1895 exited with status 1
2021-06-07 14:11:48,471||140579273841440|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 257, in get_response
    return self.kvstub.Range(
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623075108.470677444","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/solver/service.py", line 302, in run
    translated_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TRANSLATED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 283, in get
    range_response = self.get_response(key, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
    _translate_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
    raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-06-07 14:11:48,480||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 1906 exited with status 2
2021-06-07 14:12:21,665||140579273841440|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 257, in get_response
    return self.kvstub.Range(
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623075141.664867116","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/solver/service.py", line 303, in run
    solving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVING)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 283, in get
    range_response = self.get_response(key, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
    _translate_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
    raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-06-07 14:12:21,671||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 2549 exited with status 2
2021-06-07 14:15:12,482||140579273841440|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623075312.481011610","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/solver/service.py", line 302, in run
    translated_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TRANSLATED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:15:12,490||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 2630 exited with status 2
Process Process-15:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623075319.616997170","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 203, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 219, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/solver/service.py", line 271, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:15:19,623||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 3138 exited with status 1
2021-06-07 14:17:26,373||140579273841440|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 302, in get_prefix_response
    return self.kvstub.Range(
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623075446.373164717","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/solver/service.py", line 302, in run
    translated_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TRANSLATED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 318, in get_prefix
    range_response = self.get_prefix_response(key_prefix, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
    _translate_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
    raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-06-07 14:17:26,380||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 3154 exited with status 2
2021-06-07 14:17:56,877||140579273841440|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623075476.876753358","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/solver/service.py", line 302, in run
    translated_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TRANSLATED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:17:56,884||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 3476 exited with status 2
Process Process-18:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623075484.773980809","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 203, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 219, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/solver/service.py", line 271, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:18:04,785||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 3548 exited with status 1
Process Process-19:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623075491.805040904","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 203, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 219, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/solver/service.py", line 271, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:18:11,816||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 3559 exited with status 1
Process Process-20:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623075504.499138996","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 203, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 219, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/solver/service.py", line 271, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:18:24,511||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 3570 exited with status 1
2021-06-07 14:19:03,646||140579273841440|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 302, in get_prefix_response
    return self.kvstub.Range(
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623075543.645897051","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/solver/service.py", line 303, in run
    solving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVING)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 318, in get_prefix
    range_response = self.get_prefix_response(key_prefix, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
    _translate_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
    raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-06-07 14:19:03,652||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 3591 exited with status 2
2021-06-07 14:25:14,286||140579273841440|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623075914.285840827","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/solver/service.py", line 302, in run
    translated_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TRANSLATED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:25:14,293||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 3670 exited with status 2
Process Process-23:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623075921.313642216","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 203, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 219, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/solver/service.py", line 271, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:25:21,324||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 4902 exited with status 1
Process Process-24:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623075928.825120784","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 203, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 219, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/solver/service.py", line 271, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:25:28,834||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 4913 exited with status 1
Process Process-25:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623075935.850493206","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 203, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 219, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/solver/service.py", line 271, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:25:35,861||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 4929 exited with status 1
Process Process-26:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623075942.877541714","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 203, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 219, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/solver/service.py", line 271, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:25:42,888||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 4945 exited with status 1
2021-06-07 14:28:46,952||140579273841440|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623076126.952097227","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/solver/service.py", line 303, in run
    solving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVING)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:28:46,959||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 4951 exited with status 2
Process Process-28:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623076133.981031366","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 203, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 219, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/solver/service.py", line 271, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:28:53,988||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 5526 exited with status 1
2021-06-07 14:29:47,992||140579273841440|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623076187.991999097","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/solver/service.py", line 303, in run
    solving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVING)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:29:48,000||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 5539 exited with status 2
2021-06-07 14:30:39,426||140579273841440|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623076239.425434869","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/solver/service.py", line 302, in run
    translated_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TRANSLATED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:30:39,431||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 5674 exited with status 2
Process Process-31:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623076246.448139509","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 203, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 219, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/solver/service.py", line 271, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:30:46,456||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 5767 exited with status 1
Process Process-32:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623076253.476361551","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 203, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 219, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/solver/service.py", line 271, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:30:53,488||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 5783 exited with status 1
Process Process-33:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623076266.572685880","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 203, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 219, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/solver/service.py", line 271, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:31:06,583||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 5789 exited with status 1
2021-06-07 14:35:55,933||140579273841440|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623076555.931954423","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/solver/service.py", line 302, in run
    translated_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TRANSLATED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:35:55,941||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 5815 exited with status 2
2021-06-07 14:41:26,704||140579273841440|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623076886.703213811","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/solver/service.py", line 302, in run
    translated_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TRANSLATED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:41:26,711||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 6919 exited with status 2
Process Process-36:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623076904.669026317","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 203, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 219, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/solver/service.py", line 272, in setup_rpc
    target = music_messaging.Target(topic=topic)
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 103, in __init__
    self._topic_class = base.create_dynamic_model(
  File "/opt/has/conductor/conductor/common/music/model/base.py", line 186, in create_dynamic_model
    model.table_create()
  File "/opt/has/conductor/conductor/common/music/model/base.py", line 70, in table_create
    db_backend.DB_API.table_create(**kwargs)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 100, in table_create
    self.get_client().put_if_not_exists(table_key, table_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:41:44,676||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 8006 exited with status 1
2021-06-07 15:03:48,690||140579273841440|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623078228.689246936","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/solver/service.py", line 302, in run
    translated_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TRANSLATED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 15:03:48,697||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 8032 exited with status 2
Process Process-38:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623078235.964218807","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 203, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 219, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/solver/service.py", line 271, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 15:03:55,971||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 12124 exited with status 1
Process Process-39:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623078246.082704110","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 203, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 219, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/solver/service.py", line 271, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 15:04:06,089||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 12140 exited with status 1
2021-06-07 15:12:25,986||140579273841440|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 257, in get_response
    return self.kvstub.Range(
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623078745.986078691","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/solver/service.py", line 303, in run
    solving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVING)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 283, in get
    range_response = self.get_response(key, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
    _translate_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
    raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-06-07 15:12:25,994||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 12157 exited with status 2
Process Process-41:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623078755.731893272","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 203, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/solver/service.py", line 219, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/solver/service.py", line 271, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 15:12:35,738||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 13548 exited with status 1
2021-06-07 15:18:45,235||140579273841440|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623079125.233882970","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/solver/service.py", line 302, in run
    translated_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TRANSLATED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 15:18:45,242||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 13564 exited with status 2
2021-06-07 15:23:07,364||140579273841440|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 302, in get_prefix_response
    return self.kvstub.Range(
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623079387.363785469","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/solver/service.py", line 302, in run
    translated_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TRANSLATED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 318, in get_prefix
    range_response = self.get_prefix_response(key_prefix, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
    _translate_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
    raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-06-07 15:23:07,373||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 14608 exited with status 2
2021-06-07 15:24:39,501||140579273841440|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 302, in get_prefix_response
    return self.kvstub.Range(
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1623079479.500664891","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/solver/service.py", line 303, in run
    solving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVING)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 318, in get_prefix
    range_response = self.get_prefix_response(key_prefix, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
    _translate_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
    raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-06-07 15:24:39,508||140579275963168|INFO|_service_manager|cotyledon._service_manager: [-] Child 15398 exited with status 2