2022-04-08 02:22:48,480||140028408937248|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1649384568.479388137","description":"Error received from peer ipv4:10.233.76.193:2379","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/controller/translator_svc.py", line 256, in run
    fut.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/has/conductor/conductor/controller/translator_svc.py", line 201, in __check_for_templates
    template_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TEMPLATE)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 130, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 75, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 72, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 69, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2022-04-08 02:22:48,496||140028411058976|INFO|_service_manager|cotyledon._service_manager: [-] Child 201 exited with status 2
2022-04-08 03:37:33,782||140028389841696|INFO|component|conductor.common.music.messaging.component: [-] Message e0a59da1-bf57-4db1-9a3d-a246d57ea350 method plans_get received
2022-04-08 03:37:44,278||140028389841696|INFO|component|conductor.common.music.messaging.component: [-] Message e0a59da1-bf57-4db1-9a3d-a246d57ea350 method plans_get, status: completed
2022-04-08 03:37:57,307||140028389841696|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-04-08 03:39:20,980||140028408937248|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1649389160.978689723","description":"Error received from peer ipv4:10.233.76.193:2379","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/controller/translator_svc.py", line 256, in run
    fut.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/has/conductor/conductor/controller/translator_svc.py", line 201, in __check_for_templates
    template_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TEMPLATE)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 136, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 82, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 69, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2022-04-08 03:39:20,993||140028411058976|INFO|_service_manager|cotyledon._service_manager: [-] Child 5244 exited with status 2
2022-04-08 03:39:34,085||140028398328608|INFO|component|conductor.common.music.messaging.component: [-] Message 17862b30-4c60-441d-b43e-df1cad055343 method plans_get received
2022-04-08 03:39:45,182||140028398328608|INFO|component|conductor.common.music.messaging.component: [-] Message 17862b30-4c60-441d-b43e-df1cad055343 method plans_get, status: completed
2022-04-08 03:39:51,979||140028398328608|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-04-08 03:41:05,412||140028380949280|INFO|component|conductor.common.music.messaging.component: [-] Message a89bb5e0-1c76-4d37-9ef2-ce9b20dd1d68 method plans_get received
2022-04-08 03:41:14,679||140028380949280|INFO|component|conductor.common.music.messaging.component: [-] Message a89bb5e0-1c76-4d37-9ef2-ce9b20dd1d68 method plans_get, status: completed
2022-04-08 03:41:18,085||140028380949280|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-04-08 03:42:39,480||140028388780832|INFO|component|conductor.common.music.messaging.component: [-] Message 803a0238-60f3-4344-b4a4-836e73ffa26a method plans_get received
2022-04-08 03:42:54,207||140028388780832|INFO|component|conductor.common.music.messaging.component: [-] Message 803a0238-60f3-4344-b4a4-836e73ffa26a method plans_get, status: completed
2022-04-08 03:42:58,878||140028388780832|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-04-08 03:44:52,078||140028324461344|INFO|component|conductor.common.music.messaging.component: [-] Message 37744c6a-889b-4cbd-aad5-6ae9cc24db81 method plans_get received
2022-04-08 03:45:00,278||140028324461344|INFO|component|conductor.common.music.messaging.component: [-] Message 37744c6a-889b-4cbd-aad5-6ae9cc24db81 method plans_get, status: completed
2022-04-08 03:45:08,107||140028324461344|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-04-08 03:46:04,607||140028388780832|INFO|component|conductor.common.music.messaging.component: [-] Message 9c323efd-7ff0-4708-9fac-654b4064f31d method plans_get received
2022-04-08 03:46:11,981||140028388780832|INFO|component|conductor.common.music.messaging.component: [-] Message 9c323efd-7ff0-4708-9fac-654b4064f31d method plans_get, status: completed
2022-04-08 03:46:23,802||140028388780832|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-04-08 03:51:21,478||140028349123360|INFO|component|conductor.common.music.messaging.component: [-] Message 294b53be-c42f-4c6c-81d5-58cd89f41b61 method plans_get received
2022-04-08 03:51:32,501||140028349123360|INFO|component|conductor.common.music.messaging.component: [-] Message 294b53be-c42f-4c6c-81d5-58cd89f41b61 method plans_get, status: completed
2022-04-08 03:51:38,705||140028349123360|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS