By type
2021-06-07 13:58:10,946||140471217691976|INFO|prometheus_metrics|conductor.common.prometheus_metrics: [-] Prometheus metrics endpoint started at 8000
2021-06-07 13:58:10,952||140471217691976|INFO|extensions|conductor.data.plugins.inventory_provider.extensions: [-] Loaded inventory provider extensions: ['aai', 'generator']
2021-06-07 13:58:10,952||140471217691976|INFO|extensions|conductor.data.plugins.inventory_provider.extensions: [-] Initializing inventory provider extension 'aai'
2021-06-07 13:58:10,953||140471217691976|INFO|aai|conductor.data.plugins.inventory_provider.aai: [-] **** Refreshing A&AI cache *****
2021-06-07 13:58:11,242||140471217691976|ERROR|rest|conductor.common.rest: [-] Exception: (MaxRetryError("HTTPSConnectionPool(host='aai.onap', port=8443): Max retries exceeded with url: /aai/v19/cloud-infrastructure/cloud-regions/?depth=0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused'))"),)
2021-06-07 13:58:11,243||140471217691976|WARNING|rest|conductor.common.rest: [-] Retry #2/3
2021-06-07 13:58:11,533||140471217691976|ERROR|rest|conductor.common.rest: [-] Exception: (MaxRetryError("HTTPSConnectionPool(host='aai.onap', port=8443): Max retries exceeded with url: /aai/v19/cloud-infrastructure/cloud-regions/?depth=0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused'))"),)
2021-06-07 13:58:11,533||140471217691976|WARNING|rest|conductor.common.rest: [-] Retry #3/3
2021-06-07 13:58:11,822||140471217691976|ERROR|rest|conductor.common.rest: [-] Exception: (MaxRetryError("HTTPSConnectionPool(host='aai.onap', port=8443): Max retries exceeded with url: /aai/v19/cloud-infrastructure/cloud-regions/?depth=0 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused'))"),)
2021-06-07 13:58:11,823||140471217691976|ERROR|aai|conductor.data.plugins.inventory_provider.aai: [-] No response from A&AI (cloud regions: all)
2021-06-07 13:58:11,823||140471217691976|INFO|extensions|conductor.data.plugins.inventory_provider.extensions: [-] Initializing inventory provider extension 'generator'
2021-06-07 13:58:11,826||140471217691976|INFO|extensions|conductor.data.plugins.vim_controller.extensions: [-] Loaded Vim controller extensions: ['multicloud']
2021-06-07 13:58:11,827||140471217691976|INFO|extensions|conductor.data.plugins.vim_controller.extensions: [-] Initializing Vim controller extension 'multicloud'
2021-06-07 13:58:11,827||140471217691976|INFO|multicloud|conductor.data.plugins.vim_controller.multicloud: [-] **** Initializing Multicloud Vim controller *****
2021-06-07 13:58:11,828||140471217691976|INFO|extensions|conductor.data.plugins.service_controller.extensions: [-] Loaded service controller extensions: ['sdnc']
2021-06-07 13:58:11,828||140471217691976|INFO|extensions|conductor.data.plugins.service_controller.extensions: [-] Initializing service controller extension 'sdnc'
2021-06-07 13:58:40,450||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623074320.449444272","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 13:58:40,456||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 162 exited with status 2
2021-06-07 13:58:48,504||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623074328.503300191","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 13:58:48,510||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 180 exited with status 2
2021-06-07 14:01:00,239||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623074460.238417904","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:01:00,249||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 198 exited with status 2
2021-06-07 14:02:56,261||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623074576.259557883","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:02:56,269||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 572 exited with status 2
2021-06-07 14:03:06,792||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623074586.791154686","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:03:06,798||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 842 exited with status 2
2021-06-07 14:03:14,822||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623074594.821413721","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:03:14,829||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 860 exited with status 2
2021-06-07 14:03:44,366||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623074624.366138655","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:03:44,373||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 879 exited with status 2
2021-06-07 14:04:01,332||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623074641.331805281","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
return self.get_values_prefix(key_prefix, pk_name, pk_value)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
for kv in self.get_client().get_prefix(key_prefix)}
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:04:01,338||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 942 exited with status 2
2021-06-07 14:04:11,562||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
return f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 257, in get_response
return self.kvstub.Range(
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623074651.561420026","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 283, in get
range_response = self.get_response(key, **kwargs)
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
_translate_exception(exc)
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-06-07 14:04:11,570||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 969 exited with status 2
2021-06-07 14:04:22,954||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623074662.953381927","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:04:22,960||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 987 exited with status 2
2021-06-07 14:04:33,274||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623074673.272096727","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:04:33,282||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 1005 exited with status 2
2021-06-07 14:04:47,343||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623074687.341643249","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:04:47,349||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 1024 exited with status 2
2021-06-07 14:08:37,891||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623074917.890257475","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:08:37,899||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 1047 exited with status 2
2021-06-07 14:14:06,287||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623075246.286532316","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
return self.get_values_prefix(key_prefix, pk_name, pk_value)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
for kv in self.get_client().get_prefix(key_prefix)}
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:14:06,295||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 1902 exited with status 2
2021-06-07 14:14:43,877||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
return f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 257, in get_response
return self.kvstub.Range(
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623075283.875997816","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 283, in get
range_response = self.get_response(key, **kwargs)
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
_translate_exception(exc)
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-06-07 14:14:43,885||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 2966 exited with status 2
2021-06-07 14:15:04,883||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623075304.882444709","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
return self.get_values_prefix(key_prefix, pk_name, pk_value)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
for kv in self.get_client().get_prefix(key_prefix)}
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:15:04,888||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 3112 exited with status 2
2021-06-07 14:16:07,257||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623075367.256372728","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:16:07,265||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 3182 exited with status 2
2021-06-07 14:17:57,774||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623075477.773380700","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:17:57,781||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 3339 exited with status 2
2021-06-07 14:18:10,499||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623075490.498323151","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:18:10,505||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 3618 exited with status 2
2021-06-07 14:18:18,806||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623075498.805093693","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:18:18,811||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 3636 exited with status 2
2021-06-07 14:18:57,434||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623075537.433142923","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
return self.get_values_prefix(key_prefix, pk_name, pk_value)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
for kv in self.get_client().get_prefix(key_prefix)}
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:18:57,441||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 3654 exited with status 2
2021-06-07 14:25:14,825||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623075914.824444229","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:25:14,834||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 3735 exited with status 2
2021-06-07 14:25:28,314||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623075928.313555937","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:25:28,320||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 4991 exited with status 2
2021-06-07 14:25:42,332||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623075942.330647765","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:25:42,337||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 5014 exited with status 2
2021-06-07 14:28:50,712||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
return f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 302, in get_prefix_response
return self.kvstub.Range(
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623076130.711743336","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
return self.get_values_prefix(key_prefix, pk_name, pk_value)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
for kv in self.get_client().get_prefix(key_prefix)}
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 318, in get_prefix
range_response = self.get_prefix_response(key_prefix, **kwargs)
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
_translate_exception(exc)
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-06-07 14:28:50,720||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 5032 exited with status 2
2021-06-07 14:29:47,994||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623076187.993037795","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:29:48,001||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 5647 exited with status 2
2021-06-07 14:30:41,199||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623076241.199234774","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
return self.get_values_prefix(key_prefix, pk_name, pk_value)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
for kv in self.get_client().get_prefix(key_prefix)}
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:30:41,205||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 5801 exited with status 2
2021-06-07 14:30:55,381||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623076255.379229159","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:30:55,387||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 5911 exited with status 2
2021-06-07 14:35:55,784||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623076555.783507794","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:35:55,792||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 5934 exited with status 2
2021-06-07 14:41:38,598||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623076898.597285296","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:41:38,606||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 7087 exited with status 2
2021-06-07 14:59:18,533||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623077958.532571851","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 14:59:18,542||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 8181 exited with status 2
2021-06-07 15:03:58,918||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
return f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 257, in get_response
return self.kvstub.Range(
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623078238.916781069","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 283, in get
range_response = self.get_response(key, **kwargs)
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
_translate_exception(exc)
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-06-07 15:03:58,926||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 11464 exited with status 2
2021-06-07 15:04:06,951||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623078246.950134913","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 15:04:06,957||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 12297 exited with status 2
2021-06-07 15:10:22,031||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
return f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 302, in get_prefix_response
return self.kvstub.Range(
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623078622.030785285","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
return self.get_values_prefix(key_prefix, pk_name, pk_value)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
for kv in self.get_client().get_prefix(key_prefix)}
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 318, in get_prefix
range_response = self.get_prefix_response(key_prefix, **kwargs)
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
_translate_exception(exc)
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-06-07 15:10:22,040||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 12310 exited with status 2
2021-06-07 15:12:00,574||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623078720.573580437","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 15:12:00,582||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 13429 exited with status 2
2021-06-07 15:12:21,382||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623078741.381455902","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
return self.get_values_prefix(key_prefix, pk_name, pk_value)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
for kv in self.get_client().get_prefix(key_prefix)}
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 15:12:21,390||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 13655 exited with status 2
2021-06-07 15:12:35,742||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623078755.740738813","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
return self.get_values_prefix(key_prefix, pk_name, pk_value)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
for kv in self.get_client().get_prefix(key_prefix)}
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 15:12:35,749||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 13687 exited with status 2
2021-06-07 15:17:59,888||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623079079.887321244","description":"Error received from peer ipv4:10.242.207.141:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 15:17:59,896||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 13714 exited with status 2
2021-06-07 15:18:45,364||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623079125.363731356","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
return self.get_values_prefix(key_prefix, pk_name, pk_value)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
for kv in self.get_client().get_prefix(key_prefix)}
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 15:18:45,370||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 14688 exited with status 2
2021-06-07 15:23:00,364||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623079380.363337332","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
return self.get_values_prefix(key_prefix, pk_name, pk_value)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
for kv in self.get_client().get_prefix(key_prefix)}
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 15:23:00,372||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 14795 exited with status 2
2021-06-07 15:24:46,502||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
return f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 302, in get_prefix_response
return self.kvstub.Range(
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623079486.501474105","description":"Error received from peer ipv4:10.242.168.23:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
return self.get_values_prefix(key_prefix, pk_name, pk_value)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
for kv in self.get_client().get_prefix(key_prefix)}
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 318, in get_prefix
range_response = self.get_prefix_response(key_prefix, **kwargs)
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
_translate_exception(exc)
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-06-07 15:24:46,511||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 15617 exited with status 2
2021-06-07 15:24:54,553||140471149566752|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1623079494.551625670","description":"Error received from peer ipv4:10.242.80.211:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-07 15:24:54,561||140471151688480|INFO|_service_manager|cotyledon._service_manager: [-] Child 15944 exited with status 2