2022-05-09 02:53:44,187||139622998952736|INFO|component|conductor.common.music.messaging.component: [-] Message 9f2909ef-5c67-446d-a1fa-dfc55c0bda2f method plans_get received
2022-05-09 02:53:54,418||139622998952736|INFO|component|conductor.common.music.messaging.component: [-] Message 9f2909ef-5c67-446d-a1fa-dfc55c0bda2f method plans_get, status: completed
2022-05-09 02:54:03,083||139622998952736|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-05-09 02:55:25,604||139622969248544|INFO|component|conductor.common.music.messaging.component: [-] Message 1aa3cbd0-19e2-481a-adaf-7a2117e03885 method plans_get received
2022-05-09 02:55:33,112||139622969248544|INFO|component|conductor.common.music.messaging.component: [-] Message 1aa3cbd0-19e2-481a-adaf-7a2117e03885 method plans_get, status: completed
2022-05-09 02:55:40,786||139622969248544|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-05-09 02:56:56,184||139623026940704|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
    return etcd3.client(host=self.host, port=self.port,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
    return Etcd3Client(host=host,
  File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
    resp = self.auth_stub.Authenticate(auth_request, self.timeout)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "etcdserver: request timed out"
	debug_error_string = "{"created":"@1652065016.183164938","description":"Error received from peer ipv4:10.233.76.103:2379","file":"src/core/lib/surface/call.cc","file_line":1063,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 477, in run
    fut.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 334, in __check_for_messages
    self._do()
  File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 362, in _do
    _is_updated = msg.update(condition=self.enqueued_status_condition)
  File "/opt/has/conductor/conductor/common/music/messaging/message.py", line 129, in update
    return super(Message, self).update(condition)
  File "/opt/has/conductor/conductor/common/music/model/base.py", line 144, in update
    return db_backend.DB_API.row_update(**kwargs)
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 122, in row_update
    client = self.get_client()
  File "/opt/has/conductor/conductor/common/etcd/api.py", line 69, in get_client
    raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2022-05-09 02:56:56,194||139623029062432|INFO|_service_manager|cotyledon._service_manager: [-] Child 151 exited with status 2
2022-05-09 02:57:08,122||139623025879840|INFO|component|conductor.common.music.messaging.component: [-] Message 5efa6979-35fc-40ef-a827-f63d1fc13df5 method plans_get received
2022-05-09 02:57:25,211||139623025879840|INFO|component|conductor.common.music.messaging.component: [-] Message 5efa6979-35fc-40ef-a827-f63d1fc13df5 method plans_get, status: completed
2022-05-09 02:57:30,689||139623025879840|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-05-09 02:57:44,010||139623025879840|INFO|component|conductor.common.music.messaging.component: [-] Message 91bfaff1-2129-4a06-b17e-a596a5eec152 method plans_get received
2022-05-09 02:57:52,883||139623025879840|INFO|component|conductor.common.music.messaging.component: [-] Message 91bfaff1-2129-4a06-b17e-a596a5eec152 method plans_get, status: completed
2022-05-09 02:58:02,510||139623025879840|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-05-09 02:58:26,726||139623025879840|INFO|component|conductor.common.music.messaging.component: [-] Message 649de9ec-9c09-4248-b280-9b8e10ff4d33 method plans_get received
2022-05-09 02:58:35,312||139623025879840|INFO|component|conductor.common.music.messaging.component: [-] Message 649de9ec-9c09-4248-b280-9b8e10ff4d33 method plans_get, status: completed
2022-05-09 02:58:41,104||139623025879840|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-05-09 02:59:59,084||139623019109152|INFO|component|conductor.common.music.messaging.component: [-] Message 9524a1bf-4d4a-434f-bd8d-7f0fa39f74ea method plans_get received
2022-05-09 03:00:12,185||139623019109152|INFO|component|conductor.common.music.messaging.component: [-] Message 9524a1bf-4d4a-434f-bd8d-7f0fa39f74ea method plans_get, status: completed
2022-05-09 03:00:17,488||139623019109152|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-05-09 03:01:40,506||139623021230880|INFO|component|conductor.common.music.messaging.component: [-] Message 064996e9-1dc7-4625-bd3d-6cfcaae773e0 method plans_get received
2022-05-09 03:01:44,808||139623021230880|INFO|component|conductor.common.music.messaging.component: [-] Message 064996e9-1dc7-4625-bd3d-6cfcaae773e0 method plans_get, status: completed
2022-05-09 03:02:00,587||139623021230880|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-05-09 03:03:35,785||139623000013600|INFO|component|conductor.common.music.messaging.component: [-] Message 1b77980f-b835-4da6-a591-4f5c2df220a2 method plans_get received
2022-05-09 03:03:44,806||139623000013600|INFO|component|conductor.common.music.messaging.component: [-] Message 1b77980f-b835-4da6-a591-4f5c2df220a2 method plans_get, status: completed
2022-05-09 03:03:52,683||139623000013600|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-05-09 03:05:15,886||139623006378784|INFO|component|conductor.common.music.messaging.component: [-] Message 99a6bea7-6d17-4266-a42f-6774e59a37c1 method plans_get received
2022-05-09 03:05:22,386||139623006378784|INFO|component|conductor.common.music.messaging.component: [-] Message 99a6bea7-6d17-4266-a42f-6774e59a37c1 method plans_get, status: completed
2022-05-09 03:05:31,289||139623006378784|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS
2022-05-09 03:07:33,384||139623004257056|INFO|component|conductor.common.music.messaging.component: [-] Message ac7aa7ef-6fe2-49a8-92c3-11a6d10a8d15 method plans_get received
2022-05-09 03:07:42,087||139623004257056|INFO|component|conductor.common.music.messaging.component: [-] Message ac7aa7ef-6fe2-49a8-92c3-11a6d10a8d15 method plans_get, status: completed
2022-05-09 03:07:47,990||139623004257056|INFO|component|conductor.common.music.messaging.component: [-] updating the message status from working to completed, atomic update response from MUSIC SUCCESS