By type
2021-06-05 10:49:00,531||140035742240072|INFO|prometheus_metrics|conductor.common.prometheus_metrics: [-] Prometheus metrics endpoint started at 8000
2021-06-05 10:49:00,540||140035742240072|INFO|extensions|conductor.data.plugins.inventory_provider.extensions: [-] Loaded inventory provider extensions: ['aai', 'generator']
2021-06-05 10:49:00,540||140035742240072|INFO|extensions|conductor.data.plugins.inventory_provider.extensions: [-] Initializing inventory provider extension 'aai'
2021-06-05 10:49:00,541||140035742240072|INFO|aai|conductor.data.plugins.inventory_provider.aai: [-] **** Refreshing A&AI cache *****
2021-06-05 10:49:03,612||140035742240072|ERROR|rest|conductor.common.rest: [-] Exception: (MaxRetryError("HTTPSConnectionPool(host='aai.onap', port=8443): Max retries exceeded with url: /aai/v19/cloud-infrastructure/cloud-regions/?depth=0 (Caused by ConnectTimeoutError(, 'Connection to aai.onap timed out. (connect timeout=3.05)'))"),)
2021-06-05 10:49:03,613||140035742240072|WARNING|rest|conductor.common.rest: [-] Retry #2/3
2021-06-05 10:49:06,681||140035742240072|ERROR|rest|conductor.common.rest: [-] Exception: (MaxRetryError("HTTPSConnectionPool(host='aai.onap', port=8443): Max retries exceeded with url: /aai/v19/cloud-infrastructure/cloud-regions/?depth=0 (Caused by ConnectTimeoutError(, 'Connection to aai.onap timed out. (connect timeout=3.05)'))"),)
2021-06-05 10:49:06,681||140035742240072|WARNING|rest|conductor.common.rest: [-] Retry #3/3
2021-06-05 10:49:09,748||140035742240072|ERROR|rest|conductor.common.rest: [-] Exception: (MaxRetryError("HTTPSConnectionPool(host='aai.onap', port=8443): Max retries exceeded with url: /aai/v19/cloud-infrastructure/cloud-regions/?depth=0 (Caused by ConnectTimeoutError(, 'Connection to aai.onap timed out. (connect timeout=3.05)'))"),)
2021-06-05 10:49:09,749||140035742240072|ERROR|aai|conductor.data.plugins.inventory_provider.aai: [-] No response from A&AI (cloud regions: all)
2021-06-05 10:49:09,750||140035742240072|INFO|extensions|conductor.data.plugins.inventory_provider.extensions: [-] Initializing inventory provider extension 'generator'
2021-06-05 10:49:09,756||140035742240072|INFO|extensions|conductor.data.plugins.vim_controller.extensions: [-] Loaded Vim controller extensions: ['multicloud']
2021-06-05 10:49:09,756||140035742240072|INFO|extensions|conductor.data.plugins.vim_controller.extensions: [-] Initializing Vim controller extension 'multicloud'
2021-06-05 10:49:09,757||140035742240072|INFO|multicloud|conductor.data.plugins.vim_controller.multicloud: [-] **** Initializing Multicloud Vim controller *****
2021-06-05 10:49:09,758||140035742240072|INFO|extensions|conductor.data.plugins.service_controller.extensions: [-] Loaded service controller extensions: ['sdnc']
2021-06-05 10:49:09,759||140035742240072|INFO|extensions|conductor.data.plugins.service_controller.extensions: [-] Initializing service controller extension 'sdnc'
2021-06-05 11:46:47,674||140035674094368|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1622893607.672743333","description":"Error received from peer ipv4:10.242.204.88:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-05 11:46:47,711||140035676216096|INFO|_service_manager|cotyledon._service_manager: [-] Child 149 exited with status 2
2021-06-05 11:47:36,881||140035674094368|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1622893656.877923959","description":"Error received from peer ipv4:10.242.204.88:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-05 11:47:36,895||140035676216096|INFO|_service_manager|cotyledon._service_manager: [-] Child 19131 exited with status 2
2021-06-05 11:48:00,636||140035674094368|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1622893680.634495207","description":"Error received from peer ipv4:10.242.204.88:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-05 11:48:00,649||140035676216096|INFO|_service_manager|cotyledon._service_manager: [-] Child 19193 exited with status 2
2021-06-05 11:48:23,349||140035674094368|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1622893703.348358686","description":"Error received from peer ipv4:10.242.204.88:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
return self.get_values_prefix(key_prefix, pk_name, pk_value)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
for kv in self.get_client().get_prefix(key_prefix)}
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-05 11:48:23,373||140035676216096|INFO|_service_manager|cotyledon._service_manager: [-] Child 19222 exited with status 2
2021-06-05 11:51:15,228||140035674094368|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out, possibly due to connection lost"
debug_error_string = "{"created":"@1622893875.224157080","description":"Error received from peer ipv4:10.242.204.88:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out, possibly due to connection lost","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-05 11:51:15,245||140035676216096|INFO|_service_manager|cotyledon._service_manager: [-] Child 19301 exited with status 2
2021-06-05 11:53:47,270||140035674094368|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1622894027.267857822","description":"Error received from peer ipv4:10.242.204.88:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-05 11:53:47,293||140035676216096|INFO|_service_manager|cotyledon._service_manager: [-] Child 19731 exited with status 2
2021-06-05 11:59:16,071||140035674094368|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1622894356.069695855","description":"Error received from peer ipv4:10.242.204.88:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-05 11:59:16,103||140035676216096|INFO|_service_manager|cotyledon._service_manager: [-] Child 20228 exited with status 2
2021-06-05 11:59:45,969||140035674094368|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1622894385.968203853","description":"Error received from peer ipv4:10.242.204.88:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
return self.get_values_prefix(key_prefix, pk_name, pk_value)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
for kv in self.get_client().get_prefix(key_prefix)}
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-05 11:59:45,991||140035676216096|INFO|_service_manager|cotyledon._service_manager: [-] Child 21564 exited with status 2
2021-06-05 12:04:08,367||140035674094368|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 46, in handler
return f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 257, in get_response
return self.kvstub.Range(
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1622894648.364718474","description":"Error received from peer ipv4:10.242.204.88:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 283, in get
range_response = self.get_response(key, **kwargs)
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 48, in handler
_translate_exception(exc)
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 32, in _translate_exception
raise exception
etcd3.exceptions.ConnectionFailedError: etcd connection failed
2021-06-05 12:04:08,411||140035676216096|INFO|_service_manager|cotyledon._service_manager: [-] Child 21606 exited with status 2
2021-06-05 12:06:43,697||140035674094368|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out, possibly due to connection lost"
debug_error_string = "{"created":"@1622894803.692540874","description":"Error received from peer ipv4:10.242.204.88:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out, possibly due to connection lost","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
return self.get_values_prefix(key_prefix, pk_name, pk_value)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
for kv in self.get_client().get_prefix(key_prefix)}
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-05 12:06:43,721||140035676216096|INFO|_service_manager|cotyledon._service_manager: [-] Child 21924 exited with status 2
2021-06-05 12:10:51,411||140035674094368|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out, possibly due to connection lost"
debug_error_string = "{"created":"@1622895051.408008465","description":"Error received from peer ipv4:10.242.204.88:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out, possibly due to connection lost","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
return self.get_values_prefix(key_prefix, pk_name, pk_value)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
for kv in self.get_client().get_prefix(key_prefix)}
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-05 12:10:51,447||140035676216096|INFO|_service_manager|cotyledon._service_manager: [-] Child 22400 exited with status 2
2021-06-05 12:11:30,598||140035674094368|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1622895090.597622195","description":"Error received from peer ipv4:10.242.37.92:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-05 12:11:30,612||140035676216096|INFO|_service_manager|cotyledon._service_manager: [-] Child 23219 exited with status 2
2021-06-05 12:13:44,962||140035674094368|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out, possibly due to connection lost"
debug_error_string = "{"created":"@1622895224.958304889","description":"Error received from peer ipv4:10.242.204.88:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out, possibly due to connection lost","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
return self.get_values_prefix(key_prefix, pk_name, pk_value)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
for kv in self.get_client().get_prefix(key_prefix)}
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-05 12:13:44,988||140035676216096|INFO|_service_manager|cotyledon._service_manager: [-] Child 23418 exited with status 2
2021-06-05 12:15:39,080||140035674094368|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1622895339.077644694","description":"Error received from peer ipv4:10.242.204.88:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
return self.get_values_prefix(key_prefix, pk_name, pk_value)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
for kv in self.get_client().get_prefix(key_prefix)}
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-05 12:15:39,102||140035676216096|INFO|_service_manager|cotyledon._service_manager: [-] Child 23952 exited with status 2
2021-06-05 12:16:42,597||140035674094368|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1622895402.596052321","description":"Error received from peer ipv4:10.242.204.88:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 128, in row_read
schema = self.get_value(f'{keyspace}/{table}')
File "/opt/has/conductor/conductor/common/etcd/api.py", line 73, in get_value
raw_value = self.get_raw_value(key)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 70, in get_raw_value
return self.get_client().get(key)[0]
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-05 12:16:42,623||140035676216096|INFO|_service_manager|cotyledon._service_manager: [-] Child 24483 exited with status 2
2021-06-05 12:24:24,792||140035674094368|ERROR|_utils|cotyledon._utils: [-] Unhandled exception
Traceback (most recent call last):
File "/opt/has/conductor/conductor/common/etcd/api.py", line 60, in get_client
return etcd3.client(host=self.host, port=self.port,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 1178, in client
return Etcd3Client(host=host,
File "/usr/local/lib/python3.9/site-packages/etcd3/client.py", line 148, in __init__
resp = self.auth_stub.Authenticate(auth_request, self.timeout)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1622895864.788393024","description":"Error received from peer ipv4:10.242.204.88:2379","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/cotyledon/_utils.py", line 95, in exit_on_exception
yield
File "/usr/local/lib/python3.9/site-packages/cotyledon/_service.py", line 139, in _run
self.run()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 478, in run
fut.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 440, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/site-packages/futurist/_utils.py", line 49, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 336, in __check_for_messages
self._do()
File "/opt/has/conductor/conductor/common/music/messaging/component.py", line 343, in _do
msgs = self.RPC.query.all()
File "/opt/has/conductor/conductor/common/music/model/search.py", line 80, in all
rows = db_backend.DB_API.row_read(**kwargs)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 134, in row_read
return self.get_values_prefix(key_prefix, pk_name, pk_value)
File "/opt/has/conductor/conductor/common/etcd/api.py", line 80, in get_values_prefix
for kv in self.get_client().get_prefix(key_prefix)}
File "/opt/has/conductor/conductor/common/etcd/api.py", line 67, in get_client
raise EtcdClientException("Failed to establish connection with ETCD. GRPC {}".format(rpc_error.code()))
conductor.common.etcd.utils.EtcdClientException: Failed to establish connection with ETCD. GRPC StatusCode.UNAVAILABLE
2021-06-05 12:24:24,829||140035676216096|INFO|_service_manager|cotyledon._service_manager: [-] Child 24639 exited with status 2