Results

          2021-08-28 02:16:36,597|None|139731376962336|INFO|service|conductor.reservation.service:  [-] plan id: healthcheck [-] Reservation starts, changing the template status from solved to reserving, atomic update response from MUSIC SUCCESS
2021-08-28 02:16:36,597|None|139731376962336|INFO|service|conductor.reservation.service:  [-] plan id: healthcheck [-] Plan healthcheck with request id foo is reserving by machine onap-oof-has-reservation-74c5fd67d4-dd6t6. Tried to reserve it for 1 times.
2021-08-28 02:16:36,598|None|139731376962336|ERROR|service|conductor.reservation.service:  [-] plan id: healthcheck [-] no conductor_solver in translation for Plan healthcheck
2021-08-28 02:16:36,598|None|139731376962336|INFO|service|conductor.reservation.service:  [-] plan id: healthcheck [-] Plan healthcheck with request id foo Reservation complete
2021-08-28 02:16:46,101|None|139731376962336|INFO|service|conductor.reservation.service:  [-] plan id: healthcheck [-] Reservation is complete, changing the template status from reserving to done, atomic update response from MUSIC SUCCESS
2021-08-28 02:17:10,499|None|139731376962336|ERROR|_utils|cotyledon._utils:  [-] plan id: healthcheck [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630117030.498294295","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run
    solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 02:17:10,507||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 151 exited with status 2
Process Process-2:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 834, in transaction
    txn_response = self.kvstub.Txn(
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630117067.198319218","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/reservation/service.py", line 179, in setup_rpc
    target = music_messaging.Target(topic=topic)
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 103, in __init__
    self._topic_class = base.create_dynamic_model(
  File "/opt/has/conductor/conductor/common/music/model/base.py", line 186, in create_dynamic_model
    model.table_create()
  File "/opt/has/conductor/conductor/common/music/model/base.py", line 70, in table_create
    db_backend.DB_API.table_create(**kwargs)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 100, in table_create
    self.get_client().put_if_not_exists(table_key, table_value)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 444, in put_if_not_exists
    status, _ = self.transaction(
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
    _translate_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
    raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-08-28 02:17:47,206||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 486 exited with status 1
2021-08-28 02:19:52,106||139731376962336|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630117192.105438869","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run
    solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 02:19:52,113||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 536 exited with status 2
2021-08-28 02:20:26,802||139731376962336|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630117226.801314970","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run
    solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 02:20:26,808||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 725 exited with status 2
Process Process-5:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 834, in transaction
    txn_response = self.kvstub.Txn(
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630117244.599696828","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/reservation/service.py", line 179, in setup_rpc
    target = music_messaging.Target(topic=topic)
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 103, in __init__
    self._topic_class = base.create_dynamic_model(
  File "/opt/has/conductor/conductor/common/music/model/base.py", line 186, in create_dynamic_model
    model.table_create()
  File "/opt/has/conductor/conductor/common/music/model/base.py", line 70, in table_create
    db_backend.DB_API.table_create(**kwargs)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 100, in table_create
    self.get_client().put_if_not_exists(table_key, table_value)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 444, in put_if_not_exists
    status, _ = self.transaction(
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
    _translate_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
    raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-08-28 02:20:44,607||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 774 exited with status 1
2021-08-28 02:23:52,300||139731376962336|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630117432.299564114","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/reservation/service.py", line 265, in run
    reserving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.RESERVING)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 02:23:52,306||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 799 exited with status 2
2021-08-28 02:24:57,197||139731376962336|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630117497.197033979","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/reservation/service.py", line 265, in run
    reserving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.RESERVING)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 02:24:57,206||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 1089 exited with status 2
2021-08-28 02:26:49,596||139731376962336|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630117609.595999095","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/reservation/service.py", line 265, in run
    reserving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.RESERVING)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 02:26:49,603||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 1176 exited with status 2
2021-08-28 02:31:03,797||139731376962336|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630117863.796217069","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/reservation/service.py", line 265, in run
    reserving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.RESERVING)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 02:31:03,806||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 1353 exited with status 2
Process Process-10:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 834, in transaction
    txn_response = self.kvstub.Txn(
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630117878.596149136","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/reservation/service.py", line 178, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 444, in put_if_not_exists
    status, _ = self.transaction(
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
    _translate_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
    raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-08-28 02:31:18,603||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 1764 exited with status 1
2021-08-28 02:33:06,098||139731376962336|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630117986.097964789","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run
    solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 02:33:06,105||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 1786 exited with status 2
Process Process-12:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630118004.995248525","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/reservation/service.py", line 179, in setup_rpc
    target = music_messaging.Target(topic=topic)
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 103, in __init__
    self._topic_class = base.create_dynamic_model(
  File "/opt/has/conductor/conductor/common/music/model/base.py", line 186, in create_dynamic_model
    model.table_create()
  File "/opt/has/conductor/conductor/common/music/model/base.py", line 70, in table_create
    db_backend.DB_API.table_create(**kwargs)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 100, in table_create
    self.get_client().put_if_not_exists(table_key, table_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 02:33:25,003||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 1945 exited with status 1
2021-08-28 02:37:36,001||139731376962336|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630118256.001181088","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run
    solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 02:37:36,007||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 1969 exited with status 2
Process Process-14:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630118272.600114031","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/reservation/service.py", line 179, in setup_rpc
    target = music_messaging.Target(topic=topic)
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 103, in __init__
    self._topic_class = base.create_dynamic_model(
  File "/opt/has/conductor/conductor/common/music/model/base.py", line 186, in create_dynamic_model
    model.table_create()
  File "/opt/has/conductor/conductor/common/music/model/base.py", line 70, in table_create
    db_backend.DB_API.table_create(**kwargs)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 100, in table_create
    self.get_client().put_if_not_exists(table_key, table_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 02:37:52,608||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 2345 exited with status 1
Process Process-15:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 834, in transaction
    txn_response = self.kvstub.Txn(
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630118287.800461316","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/reservation/service.py", line 178, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 444, in put_if_not_exists
    status, _ = self.transaction(
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
    _translate_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
    raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-08-28 02:38:07,808||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 2371 exited with status 1
2021-08-28 02:40:23,102||139731376962336|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630118423.102346378","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/reservation/service.py", line 265, in run
    reserving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.RESERVING)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 02:40:23,109||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 2392 exited with status 2
Process Process-17:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 834, in transaction
    txn_response = self.kvstub.Txn(
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630118437.796499767","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/reservation/service.py", line 178, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 444, in put_if_not_exists
    status, _ = self.transaction(
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
    _translate_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
    raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-08-28 02:40:37,803||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 2599 exited with status 1
2021-08-28 02:41:37,801||139731376962336|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630118497.800640394","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run
    solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 02:41:37,808||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 2621 exited with status 2
2021-08-28 02:42:24,100||139731376962336|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630118544.099085935","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run
    solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 02:42:24,108||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 2718 exited with status 2
2021-08-28 02:42:57,101||139731376962336|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630118577.100398517","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run
    solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 02:42:57,109||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 2792 exited with status 2
2021-08-28 02:48:31,900||139731376962336|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630118911.899378803","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run
    solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 02:48:31,907||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 2843 exited with status 2
2021-08-28 02:49:39,296||139731376962336|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630118979.295659244","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/reservation/service.py", line 265, in run
    reserving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.RESERVING)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 02:49:39,303||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 3368 exited with status 2
2021-08-28 02:52:30,601||139731376962336|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630119150.600455274","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/reservation/service.py", line 265, in run
    reserving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.RESERVING)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 02:52:30,607||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 3465 exited with status 2
2021-08-28 02:54:18,401||139731376962336|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630119258.400769295","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run
    solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 02:54:18,408||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 3706 exited with status 2
2021-08-28 02:54:52,801||139731376962336|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630119292.800800351","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run
    solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 02:54:52,807||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 3860 exited with status 2
Process Process-26:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 834, in transaction
    txn_response = self.kvstub.Txn(
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630119324.695423621","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/reservation/service.py", line 179, in setup_rpc
    target = music_messaging.Target(topic=topic)
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 103, in __init__
    self._topic_class = base.create_dynamic_model(
  File "/opt/has/conductor/conductor/common/music/model/base.py", line 186, in create_dynamic_model
    model.table_create()
  File "/opt/has/conductor/conductor/common/music/model/base.py", line 70, in table_create
    db_backend.DB_API.table_create(**kwargs)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 100, in table_create
    self.get_client().put_if_not_exists(table_key, table_value)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 444, in put_if_not_exists
    status, _ = self.transaction(
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
    _translate_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
    raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-08-28 02:55:24,704||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 3911 exited with status 1
2021-08-28 02:56:17,099||139731376962336|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630119377.098310506","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run
    solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 02:56:17,106||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 3951 exited with status 2
2021-08-28 02:56:37,598||139731376962336|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630119397.598238199","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run
    solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 02:56:37,605||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 4026 exited with status 2
2021-08-28 02:57:58,795||139731376962336|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630119478.795263988","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/reservation/service.py", line 265, in run
    reserving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.RESERVING)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 02:57:58,802||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 4062 exited with status 2
2021-08-28 03:00:04,797||139731376962336|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630119604.796847652","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/reservation/service.py", line 264, in run
    solved_plans = self.Plan.query.get_plan_by_col("status", self.Plan.SOLVED)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 03:00:04,807||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 4193 exited with status 2
2021-08-28 03:01:53,801||139731376962336|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630119713.800839671","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/reservation/service.py", line 265, in run
    reserving_plans = self.Plan.query.get_plan_by_col("status", self.Plan.RESERVING)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 03:01:53,808||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 4366 exited with status 2
Process Process-32:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630119733.599072162","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/reservation/service.py", line 178, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-08-28 03:02:13,611||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 4517 exited with status 1
Process Process-33:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 834, in transaction
    txn_response = self.kvstub.Txn(
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1630119748.295761786","description":"Error received from peer ipv4:10.233.78.174:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/reservation/service.py", line 115, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/reservation/service.py", line 137, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/reservation/service.py", line 178, in setup_rpc
    transport = messaging.get_transport(conf=conf)
  File "/opt/has/conductor/conductor/messaging.py", line 56, in get_transport
    transport.keyspace_create(keyspace=keyspace)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 92, in keyspace_create
    self.get_client().put_if_not_exists(keyspace, "This key is a placeholder for the keyspace".encode())
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 444, in put_if_not_exists
    status, _ = self.transaction(
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
    _translate_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
    raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-08-28 03:02:28,303||139731379084064|INFO|_service_manager|cotyledon._service_manager: [-] Child 4544 exited with status 1