2022-03-05 02:17:14,115||140217056267040|INFO|component|conductor.common.music.messaging.component: [-] Message df3844ae-a7f2-4c37-a0d2-54db6729f6db method plans_get received
2022-03-05 02:17:27,209||140217056267040|INFO|component|conductor.common.music.messaging.component: [-] Message df3844ae-a7f2-4c37-a0d2-54db6729f6db method plans_get, status: completed
2022-03-05 02:17:39,607||140217056267040|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-03-05 02:19:05,049||140217086233376|INFO|component|conductor.common.music.messaging.component: [-] Message 1f0cdd34-955b-446c-bd96-fca70a6063df method plans_get received
2022-03-05 02:19:15,219||140217086233376|INFO|component|conductor.common.music.messaging.component: [-] Message 1f0cdd34-955b-446c-bd96-fca70a6063df method plans_get, status: completed
2022-03-05 02:19:24,511||140217086233376|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-03-05 02:21:00,017||140217046719264|INFO|component|conductor.common.music.messaging.component: [-] Message b80f6d8e-1662-461b-ab17-e14251d02631 method plans_get received
2022-03-05 02:21:13,711||140217046719264|INFO|component|conductor.common.music.messaging.component: [-] Message b80f6d8e-1662-461b-ab17-e14251d02631 method plans_get, status: completed
2022-03-05 02:21:24,452||140217046719264|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-03-05 02:22:26,608||140217056267040|INFO|component|conductor.common.music.messaging.component: [-] Message 1e886bf6-d859-42e1-b7db-7fe5f61e6b8c method plans_get received
2022-03-05 02:22:37,306||140217056267040|INFO|component|conductor.common.music.messaging.component: [-] Message 1e886bf6-d859-42e1-b7db-7fe5f61e6b8c method plans_get, status: completed
2022-03-05 02:22:48,208||140217056267040|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-03-05 02:23:44,607||140217078807328|INFO|component|conductor.common.music.messaging.component: [-] Message f36f6aa7-6964-405d-9498-d710384a7969 method plans_get received
2022-03-05 02:23:54,151||140217078807328|INFO|component|conductor.common.music.messaging.component: [-] Message f36f6aa7-6964-405d-9498-d710384a7969 method plans_get, status: completed
2022-03-05 02:24:00,755||140217078807328|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-03-05 02:25:36,415||140217018731296|INFO|component|conductor.common.music.messaging.component: [-] Message 7d882616-f667-40a4-b69d-8e7b46c2b999 method plans_get received
2022-03-05 02:25:50,319||140217018731296|INFO|component|conductor.common.music.messaging.component: [-] Message 7d882616-f667-40a4-b69d-8e7b46c2b999 method plans_get, status: completed
2022-03-05 02:25:59,617||140217018731296|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-03-05 02:27:44,316||140217072180000|INFO|component|conductor.common.music.messaging.component: [-] Message 6576858e-0781-475b-8cd2-bd2fc069b555 method plans_get received
2022-03-05 02:27:51,816||140217072180000|INFO|component|conductor.common.music.messaging.component: [-] Message 6576858e-0781-475b-8cd2-bd2fc069b555 method plans_get, status: completed
2022-03-05 02:28:02,109||140217072180000|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-03-05 02:32:53,809||140217048840992|INFO|component|conductor.common.music.messaging.component: [-] Message 6e4ab638-f44b-4d54-9521-3115151b30f1 method plans_get received
2022-03-05 02:33:05,180||140217048840992|INFO|component|conductor.common.music.messaging.component: [-] Message 6e4ab638-f44b-4d54-9521-3115151b30f1 method plans_get, status: completed
2022-03-05 02:33:25,413||140217048840992|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-03-05 02:52:21,110||140217091537696|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 477, in run
    fut.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 334, in __check_for_messages
    self._do()
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 341, in _do
    msgs = self.RPC.query.all()
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
    rows = db_backend.DB_API.row_read(**kwargs)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 136, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 82, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 318, in get_prefix
    range_response = self.get_prefix_response(key_prefix, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
    _translate_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 302, in get_prefix_response
    return self.kvstub.Range(
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAUTHENTICATED
	details = "etcdserver: invalid auth token"
	debug_error_string = "{"created":"@1646448741.108626851","description":"Error received from peer ipv4:10.233.75.107:2379","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"etcdserver: invalid auth token","grpc_status":16}"
>
2022-03-05 02:52:21,111||140217091537696|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/controller/translator_svc.py", line 256, in run
    fut.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/has/conductor/conductor/controller/translator_svc.py", line 202, in __check_for_templates
    translating_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TRANSLATING)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 136, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 82, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 318, in get_prefix
    range_response = self.get_prefix_response(key_prefix, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
    _translate_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 302, in get_prefix_response
    return self.kvstub.Range(
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAUTHENTICATED
	details = "etcdserver: invalid auth token"
	debug_error_string = "{"created":"@1646448741.109829571","description":"Error received from peer ipv4:10.233.75.107:2379","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"etcdserver: invalid auth token","grpc_status":16}"
>
2022-03-05 02:52:21,124||140217093659424|INFO|_service_manager|cotyledon._service_manager: [-] Child 208 exited with status 2
2022-03-05 02:52:21,140||140217093659424|INFO|_service_manager|cotyledon._service_manager: [-] Child 212 exited with status 2
2022-03-05 03:04:01,411||140217052023584|INFO|component|conductor.common.music.messaging.component: [-] Message 89d0bf21-4ebd-403e-913e-b7886acd65ba method plans_get received
2022-03-05 03:04:12,910||140217052023584|INFO|component|conductor.common.music.messaging.component: [-] Message 89d0bf21-4ebd-403e-913e-b7886acd65ba method plans_get, status: completed
2022-03-05 03:04:17,108||140217052023584|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-03-05 03:09:19,325||140217057327904|INFO|component|conductor.common.music.messaging.component: [-] Message 48431ade-5f57-4215-b3b4-3c876b54ad68 method plans_get received
2022-03-05 03:09:29,012||140217057327904|INFO|component|conductor.common.music.messaging.component: [-] Message 48431ade-5f57-4215-b3b4-3c876b54ad68 method plans_get, status: completed
2022-03-05 03:09:41,150||140217057327904|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-03-05 03:12:55,511||140217054145312|INFO|component|conductor.common.music.messaging.component: [-] Message e7b72a43-6cff-4046-8a23-860f52dd1365 method plans_get received
2022-03-05 03:13:08,450||140217054145312|INFO|component|conductor.common.music.messaging.component: [-] Message e7b72a43-6cff-4046-8a23-860f52dd1365 method plans_get, status: completed
2022-03-05 03:13:19,452||140217054145312|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-03-05 03:17:00,614||140217068997408|INFO|component|conductor.common.music.messaging.component: [-] Message 8a4aac00-586c-40c1-a005-8deb487de28a method plans_get received
2022-03-05 03:17:11,663||140217068997408|INFO|component|conductor.common.music.messaging.component: [-] Message 8a4aac00-586c-40c1-a005-8deb487de28a method plans_get, status: completed
2022-03-05 03:17:21,850||140217068997408|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-03-05 03:21:21,311||140217046719264|INFO|component|conductor.common.music.messaging.component: [-] Message 30f46004-f828-4e92-993c-5450c1c03c95 method plans_get received
2022-03-05 03:21:33,109||140217046719264|INFO|component|conductor.common.music.messaging.component: [-] Message 30f46004-f828-4e92-993c-5450c1c03c95 method plans_get, status: completed
2022-03-05 03:21:40,807||140217046719264|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-03-05 03:28:44,913||140217091537696|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1646450924.911821348","description":"Error received from peer ipv4:10.233.75.107:2379","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/controller/translator_svc.py", line 256, in run
    fut.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/has/conductor/conductor/controller/translator_svc.py", line 202, in __check_for_templates
    translating_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TRANSLATING)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 136, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 82, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 69, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2022-03-05 03:28:44,927||140217093659424|INFO|_service_manager|cotyledon._service_manager: [-] Child 11365 exited with status 2
2022-03-05 03:28:51,810||140217061571360|INFO|component|conductor.common.music.messaging.component: [-] Message 1152cec9-c55b-4931-8996-092e15e0eae1 method plans_get received
2022-03-05 03:29:03,609||140217061571360|INFO|component|conductor.common.music.messaging.component: [-] Message 1152cec9-c55b-4931-8996-092e15e0eae1 method plans_get, status: completed
2022-03-05 03:29:10,709||140217061571360|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-03-05 04:16:57,445||140217091537696|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1646453817.443997589","description":"Error received from peer ipv4:10.233.75.107:2379","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/controller/translator_svc.py", line 256, in run
    fut.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/has/conductor/conductor/controller/translator_svc.py", line 201, in __check_for_templates
    template_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TEMPLATE)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 136, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 82, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 69, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2022-03-05 04:16:57,461||140217093659424|INFO|_service_manager|cotyledon._service_manager: [-] Child 17004 exited with status 2
2022-03-05 04:16:59,707||140217091537696|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1646453819.706618626","description":"Error received from peer ipv4:10.233.75.107:2379","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 477, in run
    fut.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 334, in __check_for_messages
    self._do()
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 341, in _do
    msgs = self.RPC.query.all()
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
    rows = db_backend.DB_API.row_read(**kwargs)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 136, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 82, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 69, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2022-03-05 04:16:59,718||140217093659424|INFO|_service_manager|cotyledon._service_manager: [-] Child 11363 exited with status 2
2022-03-05 04:18:34,320||140217091537696|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1646453914.319280516","description":"Error received from peer ipv4:10.233.72.123:2379","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 477, in run
    fut.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 334, in __check_for_messages
    self._do()
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 341, in _do
    msgs = self.RPC.query.all()
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
    rows = db_backend.DB_API.row_read(**kwargs)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 130, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 75, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 72, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 69, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2022-03-05 04:18:34,334||140217093659424|INFO|_service_manager|cotyledon._service_manager: [-] Child 24200 exited with status 2
2022-03-05 04:18:41,320||140217091537696|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1646453921.319275447","description":"Error received from peer ipv4:10.233.72.123:2379","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/controller/translator_svc.py", line 256, in run
    fut.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/has/conductor/conductor/controller/translator_svc.py", line 201, in __check_for_templates
    template_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TEMPLATE)
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
    rows = db_backend.DB_API.row_read(
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 136, in row_read
    return self.get_values_prefix(key_prefix, pk_name, pk_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 82, in get_values_prefix
    for kv in self.get_client().get_prefix(key_prefix)}
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 69, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2022-03-05 04:18:41,331||140217093659424|INFO|_service_manager|cotyledon._service_manager: [-] Child 24185 exited with status 2
2022-03-05 04:18:43,082||140217091537696|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1646453923.080606948","description":"Error received from peer ipv4:10.233.71.184:2379","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 477, in run
    fut.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 334, in __check_for_messages
    self._do()
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 341, in _do
    msgs = self.RPC.query.all()
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
    rows = db_backend.DB_API.row_read(**kwargs)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 130, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 75, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 72, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 69, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2022-03-05 04:18:43,094||140217093659424|INFO|_service_manager|cotyledon._service_manager: [-] Child 24319 exited with status 2
Process Process-9:
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1646453928.614126974","description":"Error received from peer ipv4:10.233.72.123:2379","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 63, in _bootstrap_process
    target(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 161, in create_and_wait
    sw = cls(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 175, in __init__
    self.service = config.service(worker_id, *args, **kwargs)
  File "/opt/has/conductor/conductor/controller/translator_svc.py", line 78, in __init__
    self._init(conf, **kwargs)
  File "/opt/has/conductor/conductor/controller/translator_svc.py", line 88, in _init
    self.data_service = self.setup_rpc(conf, "data")
  File "/opt/has/conductor/conductor/controller/translator_svc.py", line 139, in setup_rpc
    target = music_messaging.Target(topic=topic)
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 102, in __init__
    self._topic_class = base.create_dynamic_model(
  File "/opt/has/conductor/conductor/common/music/model/base.py", line 186, in create_dynamic_model
    model.table_create()
  File "/opt/has/conductor/conductor/common/music/model/base.py", line 70, in table_create
    db_backend.DB_API.table_create(**kwargs)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 102, in table_create
    self.get_client().put_if_not_exists(table_key, table_value)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 69, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2022-03-05 04:18:48,629||140217093659424|INFO|_service_manager|cotyledon._service_manager: [-] Child 24337 exited with status 1
2022-03-05 04:18:58,956||140217091537696|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1646453938.954502807","description":"Error received from peer ipv4:10.233.72.123:2379","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 477, in run
    fut.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 334, in __check_for_messages
    self._do()
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 341, in _do
    msgs = self.RPC.query.all()
  File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
    rows = db_backend.DB_API.row_read(**kwargs)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 130, in row_read
    schema = self.get_value(f'{keyspace}/{table}')
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 75, in get_value
    raw_value = self.get_raw_value(key)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 72, in get_raw_value
    return self.get_client().get(key)[0]
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 69, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2022-03-05 04:18:58,966||140217093659424|INFO|_service_manager|cotyledon._service_manager: [-] Child 24342 exited with status 2