By type
2022-01-28 04:29:11,184||140286112471840|INFO|component|conductor.common.music.messaging.component: [-] Message 9bac59e0-6656-400a-88bb-73c45827323d method plans_get received
2022-01-28 04:29:21,137||140286112471840|INFO|component|conductor.common.music.messaging.component: [-] Message 9bac59e0-6656-400a-88bb-73c45827323d method plans_get, status: completed
2022-01-28 04:29:26,836||140286112471840|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-01-28 04:34:59,476||140286147480352|INFO|component|conductor.common.music.messaging.component: [-] Message 0e26cc64-d897-4bbe-9245-3010824d88f1 method plans_get received
2022-01-28 04:35:10,037||140286147480352|INFO|component|conductor.common.music.messaging.component: [-] Message 0e26cc64-d897-4bbe-9245-3010824d88f1 method plans_get, status: completed
2022-01-28 04:35:18,775||140286147480352|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-01-28 04:36:23,965||140286066449184|INFO|component|conductor.common.music.messaging.component: [-] Message ed72cb70-8a8e-4a40-9adb-3d10b8798151 method plans_get received
2022-01-28 04:36:36,969||140286066449184|INFO|component|conductor.common.music.messaging.component: [-] Message ed72cb70-8a8e-4a40-9adb-3d10b8798151 method plans_get, status: completed
2022-01-28 04:36:43,874||140286066449184|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-01-28 04:38:03,185||140286108228384|INFO|component|conductor.common.music.messaging.component: [-] Message 6bca1a6c-77f9-4700-b0d4-164eb1f32e0e method plans_get received
2022-01-28 04:38:17,737||140286108228384|INFO|component|conductor.common.music.messaging.component: [-] Message 6bca1a6c-77f9-4700-b0d4-164eb1f32e0e method plans_get, status: completed
2022-01-28 04:38:28,932||140286108228384|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-01-28 04:41:02,034||140286098680608|INFO|component|conductor.common.music.messaging.component: [-] Message 0f1b2960-25e7-4247-8df5-68a0839fe464 method plans_get received
2022-01-28 04:41:12,577||140286098680608|INFO|component|conductor.common.music.messaging.component: [-] Message 0f1b2960-25e7-4247-8df5-68a0839fe464 method plans_get, status: completed
2022-01-28 04:41:18,178||140286098680608|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-01-28 04:43:44,175||140286111410976|INFO|component|conductor.common.music.messaging.component: [-] Message a9195114-bdc0-45c2-bd14-966a6ad1a0a2 method plans_get received
2022-01-28 04:43:54,274||140286111410976|INFO|component|conductor.common.music.messaging.component: [-] Message a9195114-bdc0-45c2-bd14-966a6ad1a0a2 method plans_get, status: completed
2022-01-28 04:43:59,974||140286111410976|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-01-28 04:49:11,872||140286099741472|INFO|component|conductor.common.music.messaging.component: [-] Message 4be9c243-3733-45c4-b9d4-960d5e8037ad method plans_get received
2022-01-28 04:49:19,231||140286099741472|INFO|component|conductor.common.music.messaging.component: [-] Message 4be9c243-3733-45c4-b9d4-960d5e8037ad method plans_get, status: completed
2022-01-28 04:49:23,771||140286099741472|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-01-28 05:09:28,130||140286149458720|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1643346568.129472932","description":"Error received from peer ipv4:10.233.74.104:2379","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/controller/translator_svc.py", line 256, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/controller/translator_svc.py", line 202, in __check_for_templates
translating_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TRANSLATING)
File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
rows = db_backend.DB_API.row_read(
File "/opt/has/conductor/conductor/common/etcd/api.py", line 130, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 75, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 72, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 69, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2022-01-28 05:09:28,140||140286151985952|INFO|_service_manager|cotyledon._service_manager: [-] Child 212 exited with status 2
2022-01-28 06:00:07,466||140286149602080|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1643349607.465466708","description":"Error received from peer ipv4:10.233.76.221:2379","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/controller/translator_svc.py", line 256, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/controller/translator_svc.py", line 202, in __check_for_templates
translating_plans = self.Plan.query.get_plan_by_col("status", self.Plan.TRANSLATING)
File "/opt/has/conductor/conductor/common/music/model/search.py", line 87, in get_plan_by_col
rows = db_backend.DB_API.row_read(
File "/opt/has/conductor/conductor/common/etcd/api.py", line 130, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 75, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 72, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 69, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2022-01-28 06:00:07,476||140286151985952|INFO|_service_manager|cotyledon._service_manager: [-] Child 16424 exited with status 2
2022-01-28 07:42:17,972||140286149864224|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1643355737.972020623","description":"Error received from peer ipv4:10.233.76.221:2379","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 477, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 334, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 341, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 136, in row_read
return self.get_values_prefix(key_prefix, pk_name, pk_value)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 82, in get_values_prefix
for kv in self.get_client().get_prefix(key_prefix)}
File "/opt/has/conductor/conductor/common/etcd/api.py", line 69, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2022-01-28 07:42:17,982||140286151985952|INFO|_service_manager|cotyledon._service_manager: [-] Child 208 exited with status 2
2022-01-28 08:24:23,866||140286149864224|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1643358263.864753320","description":"Error received from peer ipv4:10.233.76.221:2379","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 477, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 334, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 341, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 136, in row_read
return self.get_values_prefix(key_prefix, pk_name, pk_value)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 82, in get_values_prefix
for kv in self.get_client().get_prefix(key_prefix)}
File "/opt/has/conductor/conductor/common/etcd/api.py", line 69, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2022-01-28 08:24:23,877||140286151985952|INFO|_service_manager|cotyledon._service_manager: [-] Child 7573 exited with status 2
2022-01-28 09:24:33,275||140286114593568|INFO|component|conductor.common.music.messaging.component: [-] Message 68b253d7-dc8b-4fb0-97df-b71823c54b3e method plans_get received
2022-01-28 09:24:43,531||140286114593568|INFO|component|conductor.common.music.messaging.component: [-] Message 68b253d7-dc8b-4fb0-97df-b71823c54b3e method plans_get, status: completed
2022-01-28 09:24:52,675||140286114593568|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-01-28 09:25:45,974||140286130506528|INFO|component|conductor.common.music.messaging.component: [-] Message 8f0e0ee3-c401-440b-beed-4e0106341574 method plans_get received
2022-01-28 09:25:58,170||140286130506528|INFO|component|conductor.common.music.messaging.component: [-] Message 8f0e0ee3-c401-440b-beed-4e0106341574 method plans_get, status: completed
2022-01-28 09:26:15,273||140286130506528|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-01-28 09:27:36,683||140286136871712|INFO|component|conductor.common.music.messaging.component: [-] Message 907fa616-9ead-478a-8068-7841e3a8fde6 method plans_get received
2022-01-28 09:27:46,135||140286136871712|INFO|component|conductor.common.music.messaging.component: [-] Message 907fa616-9ead-478a-8068-7841e3a8fde6 method plans_get, status: completed
2022-01-28 09:28:00,672||140286136871712|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-01-28 09:28:23,774||140286114593568|INFO|component|conductor.common.music.messaging.component: [-] Message 68a68b05-cd42-42ad-9418-cea707bd8439 method plans_get received
2022-01-28 09:28:27,878||140286114593568|INFO|component|conductor.common.music.messaging.component: [-] Message 68a68b05-cd42-42ad-9418-cea707bd8439 method plans_get, status: completed
2022-01-28 09:28:39,465||140286114593568|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-01-28 09:29:00,874||140286071896864|INFO|component|conductor.common.music.messaging.component: [-] Message 439fca6e-b597-441f-8ede-4a63502cea67 method plans_get received
2022-01-28 09:29:06,530||140286071896864|INFO|component|conductor.common.music.messaging.component: [-] Message 439fca6e-b597-441f-8ede-4a63502cea67 method plans_get, status: completed
2022-01-28 09:29:11,031||140286071896864|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-01-28 09:30:25,632||140286114593568|INFO|component|conductor.common.music.messaging.component: [-] Message d3d7eaab-dcb0-4fd2-aab7-394c18d1e3d7 method plans_get received
2022-01-28 09:30:34,131||140286114593568|INFO|component|conductor.common.music.messaging.component: [-] Message d3d7eaab-dcb0-4fd2-aab7-394c18d1e3d7 method plans_get, status: completed
2022-01-28 09:30:39,473||140286114593568|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-01-28 09:32:13,781||140286110350112|INFO|component|conductor.common.music.messaging.component: [-] Message 1e6bb84f-552e-4b41-a436-eaaaeb042712 method plans_get received
2022-01-28 09:32:25,472||140286110350112|INFO|component|conductor.common.music.messaging.component: [-] Message 1e6bb84f-552e-4b41-a436-eaaaeb042712 method plans_get, status: completed
2022-01-28 09:32:30,677||140286110350112|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-01-28 09:34:29,274||140286132628256|INFO|component|conductor.common.music.messaging.component: [-] Message 61d8e859-df87-4cd5-80b8-b9d740ef077f method plans_get received
2022-01-28 09:34:38,965||140286132628256|INFO|component|conductor.common.music.messaging.component: [-] Message 61d8e859-df87-4cd5-80b8-b9d740ef077f method plans_get, status: completed
2022-01-28 09:34:46,531||140286132628256|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-01-28 09:38:20,933||140286064470816|INFO|component|conductor.common.music.messaging.component: [-] Message 079e33c6-ac0b-470a-8d3c-4650c8cffc2b method plans_get received
2022-01-28 09:38:29,777||140286064470816|INFO|component|conductor.common.music.messaging.component: [-] Message 079e33c6-ac0b-470a-8d3c-4650c8cffc2b method plans_get, status: completed
2022-01-28 09:38:37,777||140286064470816|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-01-28 09:43:39,330||140286087809824|INFO|component|conductor.common.music.messaging.component: [-] Message 4439d568-3152-4c3a-abc4-c661bdeed236 method plans_get received
2022-01-28 09:43:47,093||140286087809824|INFO|component|conductor.common.music.messaging.component: [-] Message 4439d568-3152-4c3a-abc4-c661bdeed236 method plans_get, status: completed
2022-01-28 09:43:50,531||140286087809824|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS