2021-07-13T00:00:02.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bcdb17d1-07d0-4a3e-ab14-61f490ae694b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:00:02.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bcdb17d1-07d0-4a3e-ab14-61f490ae694b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@14f97856 2021-07-13T00:00:02.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bcdb17d1-07d0-4a3e-ab14-61f490ae694b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:00:02.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bcdb17d1-07d0-4a3e-ab14-61f490ae694b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:00:03.961Z [qtp817348612-105] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "Forwarding" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "71216339-9e4c-4685-a4ed-1aa0b1403dbf" INVARIANT_UUID = "6bd01999-8fc2-4940-8b19-15cad8b956ea" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "" CURR_STATE = "CERTIFIED" STATUS = "201" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.Forwarding" RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=71216339-9e4c-4685-a4ed-1aa0b1403dbf EntryTimestamp=2021-07-12T23:59:50.569Z InvocationID=0e86beae-2757-40e1-9de7-4a0449ba3bc7 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:00:03.962Z [qtp817348612-105] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "Forwarding" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "71216339-9e4c-4685-a4ed-1aa0b1403dbf" INVARIANT_UUID = "6bd01999-8fc2-4940-8b19-15cad8b956ea" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "" CURR_STATE = "CERTIFIED" STATUS = "201" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.Forwarding" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=71216339-9e4c-4685-a4ed-1aa0b1403dbf TargetServiceName=null EntryTimestamp=2021-07-12T23:59:50.569Z InvocationID=0e86beae-2757-40e1-9de7-4a0449ba3bc7 ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:00:05.370Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=80490b19-33a0-4132-aa57-79f58538f41e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:00:05.370Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=80490b19-33a0-4132-aa57-79f58538f41e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@29b4cf5e 2021-07-13T00:00:05.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=80490b19-33a0-4132-aa57-79f58538f41e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:00:05.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=80490b19-33a0-4132-aa57-79f58538f41e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:00:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:00:07.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:00:08.814Z [qtp817348612-105] INFO o.o.s.b.m.o.impl.GraphLockOperation AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "Forwarding" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "71216339-9e4c-4685-a4ed-1aa0b1403dbf" INVARIANT_UUID = "6bd01999-8fc2-4940-8b19-15cad8b956ea" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "" CURR_STATE = "CERTIFIED" STATUS = "201" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.Forwarding" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=71216339-9e4c-4685-a4ed-1aa0b1403dbf TargetServiceName=null EntryTimestamp=2021-07-12T23:59:50.569Z InvocationID=0e86beae-2757-40e1-9de7-4a0449ba3bc7 ErrorCode=500 lock resource with id 1b8cff82-2b55-4752-8fe4-dcba34694095 2021-07-13T00:00:09.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ca0b2c6-c3ac-40b2-b0d5-f3a0ff4181e1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:00:09.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ca0b2c6-c3ac-40b2-b0d5-f3a0ff4181e1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@333c38ba 2021-07-13T00:00:09.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ca0b2c6-c3ac-40b2-b0d5-f3a0ff4181e1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:00:09.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ca0b2c6-c3ac-40b2-b0d5-f3a0ff4181e1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:00:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:00:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:00:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:00:11.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1adea3b8-90ab-41c1-866e-d05a39e832bb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:00:11.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1adea3b8-90ab-41c1-866e-d05a39e832bb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@20fe7f1e 2021-07-13T00:00:11.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1adea3b8-90ab-41c1-866e-d05a39e832bb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:00:11.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1adea3b8-90ab-41c1-866e-d05a39e832bb PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:00:12.865Z [qtp817348612-105] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "Forwarding" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "71216339-9e4c-4685-a4ed-1aa0b1403dbf" INVARIANT_UUID = "6bd01999-8fc2-4940-8b19-15cad8b956ea" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "1b8cff82-2b55-4752-8fe4-dcba34694095.assettoscatemplate" CURR_ARTIFACT_UUID = "1b8cff82-2b55-4752-8fe4-dcba34694095.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-Forwarding-template.yml,0,1b8cff82-2b55-4752-8fe4-dcba34694095.assettoscatemplate,0 " STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=082c28b4-878e-4145-a6cf-0155b626c5ba TargetServiceName=null EntryTimestamp=2021-07-12T23:59:50.569Z InvocationID=0e86beae-2757-40e1-9de7-4a0449ba3bc7 ErrorCode=500 Starting to create Csar for component Forwarding 2021-07-13T00:00:13.157Z [qtp817348612-105] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "Forwarding" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "71216339-9e4c-4685-a4ed-1aa0b1403dbf" INVARIANT_UUID = "6bd01999-8fc2-4940-8b19-15cad8b956ea" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "1b8cff82-2b55-4752-8fe4-dcba34694095.assettoscatemplate" CURR_ARTIFACT_UUID = "1b8cff82-2b55-4752-8fe4-dcba34694095.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-Forwarding-template.yml,0,1b8cff82-2b55-4752-8fe4-dcba34694095.assettoscatemplate,0 " STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=082c28b4-878e-4145-a6cf-0155b626c5ba TargetServiceName=null EntryTimestamp=2021-07-12T23:59:50.569Z InvocationID=0e86beae-2757-40e1-9de7-4a0449ba3bc7 ErrorCode=500 Ended create Csar for component Forwarding 2021-07-13T00:00:13.157Z [qtp817348612-105] INFO o.o.s.b.m.o.impl.GraphLockOperation AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "Forwarding" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "71216339-9e4c-4685-a4ed-1aa0b1403dbf" INVARIANT_UUID = "6bd01999-8fc2-4940-8b19-15cad8b956ea" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "1b8cff82-2b55-4752-8fe4-dcba34694095.assettoscatemplate" CURR_ARTIFACT_UUID = "1b8cff82-2b55-4752-8fe4-dcba34694095.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-Forwarding-template.yml,0,1b8cff82-2b55-4752-8fe4-dcba34694095.assettoscatemplate,0 " STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=082c28b4-878e-4145-a6cf-0155b626c5ba TargetServiceName=null EntryTimestamp=2021-07-12T23:59:50.569Z InvocationID=0e86beae-2757-40e1-9de7-4a0449ba3bc7 ErrorCode=500 lock resource with id 1b8cff82-2b55-4752-8fe4-dcba34694095 2021-07-13T00:00:15.267Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d029035-00cd-426d-a7db-e5069e3382dd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:00:15.267Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d029035-00cd-426d-a7db-e5069e3382dd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f9a989f 2021-07-13T00:00:15.267Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d029035-00cd-426d-a7db-e5069e3382dd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:00:15.267Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d029035-00cd-426d-a7db-e5069e3382dd PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:00:16.373Z [qtp817348612-105] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "Forwarding" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "71216339-9e4c-4685-a4ed-1aa0b1403dbf" INVARIANT_UUID = "6bd01999-8fc2-4940-8b19-15cad8b956ea" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "1b8cff82-2b55-4752-8fe4-dcba34694095.assettoscacsar" CURR_ARTIFACT_UUID = "1b8cff82-2b55-4752-8fe4-dcba34694095.assettoscacsar" ARTIFACT_DATA = "TOSCA,'assettoscacsar',TOSCA_CSAR,resource-Forwarding-csar.csar,0,1b8cff82-2b55-4752-8fe4-dcba34694095.assettoscacsar,0 " STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR ResponseCode=201 RequestID=null ElapsedTime=25804 PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp StatusCode=COMPLETE TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=ccf3a6f4-1eee-4011-95f6-8330eb7d9ef9 TargetServiceName=null EntryTimestamp=2021-07-12T23:59:50.569Z InvocationID=0e86beae-2757-40e1-9de7-4a0449ba3bc7 ErrorCode=500 LogTimestamp=2021-07-13T00:00:16.373107Z ResponseDesc=Created Exiting. 2021-07-13T00:00:16.374Z [qtp817348612-105] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 SC="201" 2021-07-13T00:00:16.730Z [qtp817348612-18] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=cb93d98d-ff4f-4b3a-b9a7-535f9bb2712b InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS RemoteHost=10.242.77.79 ServiceName=v1/catalog/upload/multipart ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:00:16.729840Z InvocationID=037da74a-52eb-4511-ae63-cba950074d9e LogTimestamp=2021-07-13T00:00:16.729890Z Entering 2021-07-13T00:00:16.730Z [qtp817348612-18] INFO o.o.sdc.be.filters.BeServletFilter RequestID=cb93d98d-ff4f-4b3a-b9a7-535f9bb2712b InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS userId=jh0003 RemoteHost=10.242.77.79 ServiceName=v1/catalog/upload/multipart ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:00:16.729840Z InvocationID=037da74a-52eb-4511-ae63-cba950074d9e LogTimestamp=2021-07-13T00:00:16.729890Z localAddr=10.242.98.212 remoteAddr=10.242.77.79 POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 2021-07-13T00:00:18.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17521752-4385-46da-aa5c-1e8199b8ba4b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:00:18.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17521752-4385-46da-aa5c-1e8199b8ba4b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@70db760d 2021-07-13T00:00:18.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17521752-4385-46da-aa5c-1e8199b8ba4b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:00:18.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17521752-4385-46da-aa5c-1e8199b8ba4b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:00:20.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ecdc8672-16bd-4e79-8b1e-28d8dcc4b215 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:00:20.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ecdc8672-16bd-4e79-8b1e-28d8dcc4b215 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@22f06944 2021-07-13T00:00:20.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ecdc8672-16bd-4e79-8b1e-28d8dcc4b215 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:00:20.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ecdc8672-16bd-4e79-8b1e-28d8dcc4b215 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:00:22.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:00:22.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:00:23.013Z [qtp817348612-18] INFO o.o.s.b.m.o.impl.GraphLockOperation RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:00:16.730Z InvocationID=980d7a75-ade1-40a1-b7c9-06ad94eadbfe PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp lock resource with id 3f9da222-5c0a-403e-a085-cd37e81def66 2021-07-13T00:00:23.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a743714a-1554-450b-8963-8a38e41170a1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:00:23.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a743714a-1554-450b-8963-8a38e41170a1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@71a5f48b 2021-07-13T00:00:23.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a743714a-1554-450b-8963-8a38e41170a1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:00:23.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a743714a-1554-450b-8963-8a38e41170a1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:00:24.349Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:00:26.466Z [qtp817348612-18] INFO o.o.s.b.c.l.LifecycleBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:00:16.730Z InvocationID=980d7a75-ade1-40a1-b7c9-06ad94eadbfe PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp owner of resource 3f9da222-5c0a-403e-a085-cd37e81def66 is jh0003 2021-07-13T00:00:27.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56723b55-a37f-46a4-96b7-6bc52359a44d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:00:27.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56723b55-a37f-46a4-96b7-6bc52359a44d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5ad0d440 2021-07-13T00:00:27.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56723b55-a37f-46a4-96b7-6bc52359a44d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:00:27.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56723b55-a37f-46a4-96b7-6bc52359a44d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:00:27.614Z [qtp817348612-18] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VDU Compute" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "b525ed79-a6fe-426a-a468-e26b59a5af52" INVARIANT_UUID = "da95a1a0-6361-4d21-848b-530cf494cb9e" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:00:16.730Z InvocationID=980d7a75-ade1-40a1-b7c9-06ad94eadbfe PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:00:27.614Z [qtp817348612-18] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VDU Compute" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "b525ed79-a6fe-426a-a468-e26b59a5af52" INVARIANT_UUID = "da95a1a0-6361-4d21-848b-530cf494cb9e" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:00:16.730Z InvocationID=980d7a75-ade1-40a1-b7c9-06ad94eadbfe ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:00:29.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c26bbc66-db69-468c-a5e2-3c796f6c27e4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:00:29.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c26bbc66-db69-468c-a5e2-3c796f6c27e4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7110ba09 2021-07-13T00:00:29.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c26bbc66-db69-468c-a5e2-3c796f6c27e4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:00:29.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c26bbc66-db69-468c-a5e2-3c796f6c27e4 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:00:33.170Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e104e6f-d1dc-4389-a4e7-6c8ad5067ac8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:00:33.171Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e104e6f-d1dc-4389-a4e7-6c8ad5067ac8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@508b4726 2021-07-13T00:00:33.171Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e104e6f-d1dc-4389-a4e7-6c8ad5067ac8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:00:33.171Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e104e6f-d1dc-4389-a4e7-6c8ad5067ac8 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:00:34.077Z [qtp817348612-18] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VDU Compute" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "b525ed79-a6fe-426a-a468-e26b59a5af52" INVARIANT_UUID = "da95a1a0-6361-4d21-848b-530cf494cb9e" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=f898bf2d-367f-477b-b4bf-b4d03036b0d6 TargetServiceName=null EntryTimestamp=2021-07-13T00:00:16.730Z InvocationID=980d7a75-ade1-40a1-b7c9-06ad94eadbfe ErrorCode=500 The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:00:34.078Z [qtp817348612-18] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VDU Compute" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "b525ed79-a6fe-426a-a468-e26b59a5af52" INVARIANT_UUID = "da95a1a0-6361-4d21-848b-530cf494cb9e" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=f898bf2d-367f-477b-b4bf-b4d03036b0d6 TargetServiceName=null EntryTimestamp=2021-07-13T00:00:16.730Z InvocationID=980d7a75-ade1-40a1-b7c9-06ad94eadbfe ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:00:36.468Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea3e723d-88f2-4207-9afb-04b49a652ba0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:00:36.468Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea3e723d-88f2-4207-9afb-04b49a652ba0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e54e010 2021-07-13T00:00:36.468Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea3e723d-88f2-4207-9afb-04b49a652ba0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:00:36.468Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea3e723d-88f2-4207-9afb-04b49a652ba0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:00:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:00:37.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:00:38.774Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6166b14a-d62b-4c99-bb54-c98fab937ff9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:00:38.774Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6166b14a-d62b-4c99-bb54-c98fab937ff9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@527a574e 2021-07-13T00:00:38.774Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6166b14a-d62b-4c99-bb54-c98fab937ff9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:00:38.774Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6166b14a-d62b-4c99-bb54-c98fab937ff9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:00:41.561Z [qtp817348612-18] INFO o.o.s.b.m.o.impl.GraphLockOperation AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VDU Compute" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "b525ed79-a6fe-426a-a468-e26b59a5af52" INVARIANT_UUID = "da95a1a0-6361-4d21-848b-530cf494cb9e" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" STATUS = "200" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.Vdu.Compute" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=f898bf2d-367f-477b-b4bf-b4d03036b0d6 TargetServiceName=null EntryTimestamp=2021-07-13T00:00:16.730Z InvocationID=980d7a75-ade1-40a1-b7c9-06ad94eadbfe ErrorCode=500 lock resource with id f91eb779-1810-4803-a317-7e11486ea226 2021-07-13T00:00:41.778Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53647361-2871-4745-b1a5-e5ca06e9ec8e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:00:41.779Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53647361-2871-4745-b1a5-e5ca06e9ec8e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@264e7ffa 2021-07-13T00:00:41.779Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53647361-2871-4745-b1a5-e5ca06e9ec8e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:00:41.779Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53647361-2871-4745-b1a5-e5ca06e9ec8e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:00:42.862Z [qtp817348612-18] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VDU Compute" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "b525ed79-a6fe-426a-a468-e26b59a5af52" INVARIANT_UUID = "da95a1a0-6361-4d21-848b-530cf494cb9e" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" STATUS = "200" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.Vdu.Compute" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=f898bf2d-367f-477b-b4bf-b4d03036b0d6 TargetServiceName=null EntryTimestamp=2021-07-13T00:00:16.730Z InvocationID=980d7a75-ade1-40a1-b7c9-06ad94eadbfe ErrorCode=500 owner of resource f91eb779-1810-4803-a317-7e11486ea226 is jh0003 2021-07-13T00:00:45.168Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5bfb0e9c-7d9d-4002-a9fd-a94e555fada7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:00:45.168Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5bfb0e9c-7d9d-4002-a9fd-a94e555fada7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2cf6be62 2021-07-13T00:00:45.168Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5bfb0e9c-7d9d-4002-a9fd-a94e555fada7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:00:45.168Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5bfb0e9c-7d9d-4002-a9fd-a94e555fada7 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:00:47.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4facf04c-7a60-4e2f-9e8d-82165affc36c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:00:47.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4facf04c-7a60-4e2f-9e8d-82165affc36c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@58b8f8c 2021-07-13T00:00:47.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4facf04c-7a60-4e2f-9e8d-82165affc36c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:00:47.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4facf04c-7a60-4e2f-9e8d-82165affc36c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:00:50.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e25081fc-1464-4941-bad0-8f0ba035c72b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:00:50.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e25081fc-1464-4941-bad0-8f0ba035c72b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@54b6854 2021-07-13T00:00:50.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e25081fc-1464-4941-bad0-8f0ba035c72b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:00:50.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e25081fc-1464-4941-bad0-8f0ba035c72b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:00:52.270Z [qtp817348612-18] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VDU Compute" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "f898bf2d-367f-477b-b4bf-b4d03036b0d6" INVARIANT_UUID = "da95a1a0-6361-4d21-848b-530cf494cb9e" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=f898bf2d-367f-477b-b4bf-b4d03036b0d6 TargetServiceName=null EntryTimestamp=2021-07-13T00:00:16.730Z InvocationID=980d7a75-ade1-40a1-b7c9-06ad94eadbfe ErrorCode=500 owner of resource f91eb779-1810-4803-a317-7e11486ea226 is jh0003 2021-07-13T00:00:52.271Z [qtp817348612-18] INFO o.o.s.b.c.l.CertificationChangeTransition AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VDU Compute" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "f898bf2d-367f-477b-b4bf-b4d03036b0d6" INVARIANT_UUID = "da95a1a0-6361-4d21-848b-530cf494cb9e" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=f898bf2d-367f-477b-b4bf-b4d03036b0d6 TargetServiceName=null EntryTimestamp=2021-07-13T00:00:16.730Z InvocationID=980d7a75-ade1-40a1-b7c9-06ad94eadbfe ErrorCode=500 validate before certification change. resource name=VDU Compute, oldState=NOT_CERTIFIED_CHECKIN, owner userId=jh0003 2021-07-13T00:00:52.271Z [qtp817348612-18] INFO o.o.s.b.c.l.CertificationChangeTransition AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VDU Compute" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "f898bf2d-367f-477b-b4bf-b4d03036b0d6" INVARIANT_UUID = "da95a1a0-6361-4d21-848b-530cf494cb9e" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=f898bf2d-367f-477b-b4bf-b4d03036b0d6 TargetServiceName=null EntryTimestamp=2021-07-13T00:00:16.730Z InvocationID=980d7a75-ade1-40a1-b7c9-06ad94eadbfe ErrorCode=500 start performing certification change for resource f91eb779-1810-4803-a317-7e11486ea226 2021-07-13T00:00:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:00:52.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:00:53.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2382453-d5f9-4318-a613-20eec41f5435 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:00:53.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2382453-d5f9-4318-a613-20eec41f5435 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@50af76a8 2021-07-13T00:00:53.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2382453-d5f9-4318-a613-20eec41f5435 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:00:53.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2382453-d5f9-4318-a613-20eec41f5435 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:00:56.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af38b090-642f-43dd-a439-812d8fe25a8f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:00:56.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af38b090-642f-43dd-a439-812d8fe25a8f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4933d771 2021-07-13T00:00:56.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af38b090-642f-43dd-a439-812d8fe25a8f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:00:56.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af38b090-642f-43dd-a439-812d8fe25a8f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:00:58.421Z [qtp817348612-18] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VDU Compute" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "f898bf2d-367f-477b-b4bf-b4d03036b0d6" INVARIANT_UUID = "da95a1a0-6361-4d21-848b-530cf494cb9e" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "3f9da222-5c0a-403e-a085-cd37e81def66.assettoscatemplate" CURR_ARTIFACT_UUID = "3f9da222-5c0a-403e-a085-cd37e81def66.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VduCompute-template.yml,0,3f9da222-5c0a-403e-a085-cd37e81def66.assettoscatemplate,1,41687cee-63fe-4210-ac3c-30acf233ce53" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=a0a712bd-b405-43e5-ab12-930deac76a5d TargetServiceName=null EntryTimestamp=2021-07-13T00:00:16.730Z InvocationID=980d7a75-ade1-40a1-b7c9-06ad94eadbfe ErrorCode=500 Starting to create Csar for component VDU Compute 2021-07-13T00:00:58.718Z [qtp817348612-18] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VDU Compute" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "f898bf2d-367f-477b-b4bf-b4d03036b0d6" INVARIANT_UUID = "da95a1a0-6361-4d21-848b-530cf494cb9e" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "3f9da222-5c0a-403e-a085-cd37e81def66.assettoscatemplate" CURR_ARTIFACT_UUID = "3f9da222-5c0a-403e-a085-cd37e81def66.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VduCompute-template.yml,0,3f9da222-5c0a-403e-a085-cd37e81def66.assettoscatemplate,1,41687cee-63fe-4210-ac3c-30acf233ce53" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=a0a712bd-b405-43e5-ab12-930deac76a5d TargetServiceName=null EntryTimestamp=2021-07-13T00:00:16.730Z InvocationID=980d7a75-ade1-40a1-b7c9-06ad94eadbfe ErrorCode=500 Ended create Csar for component VDU Compute 2021-07-13T00:01:00.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb0f2528-0ed4-4aa6-93c3-3ddc1df8a105 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:01:00.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb0f2528-0ed4-4aa6-93c3-3ddc1df8a105 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@109dd634 2021-07-13T00:01:00.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb0f2528-0ed4-4aa6-93c3-3ddc1df8a105 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:01:00.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb0f2528-0ed4-4aa6-93c3-3ddc1df8a105 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:01:02.873Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28707b9f-8203-42a2-977a-f51b516cfcb8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:01:02.873Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28707b9f-8203-42a2-977a-f51b516cfcb8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4eca05bb 2021-07-13T00:01:02.873Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28707b9f-8203-42a2-977a-f51b516cfcb8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:01:02.873Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28707b9f-8203-42a2-977a-f51b516cfcb8 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:01:04.815Z [qtp817348612-18] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU Compute" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "f898bf2d-367f-477b-b4bf-b4d03036b0d6" INVARIANT_UUID = "da95a1a0-6361-4d21-848b-530cf494cb9e" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=2292e49c-0404-49f1-98f8-b265e70d3a42 TargetServiceName=null EntryTimestamp=2021-07-13T00:00:16.730Z InvocationID=980d7a75-ade1-40a1-b7c9-06ad94eadbfe ErrorCode=500 The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:01:04.816Z [qtp817348612-18] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU Compute" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "f898bf2d-367f-477b-b4bf-b4d03036b0d6" INVARIANT_UUID = "da95a1a0-6361-4d21-848b-530cf494cb9e" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=2292e49c-0404-49f1-98f8-b265e70d3a42 TargetServiceName=null EntryTimestamp=2021-07-13T00:00:16.730Z InvocationID=980d7a75-ade1-40a1-b7c9-06ad94eadbfe ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:01:04.816Z [qtp817348612-18] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU Compute" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "f898bf2d-367f-477b-b4bf-b4d03036b0d6" INVARIANT_UUID = "da95a1a0-6361-4d21-848b-530cf494cb9e" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=2292e49c-0404-49f1-98f8-b265e70d3a42 TargetServiceName=null EntryTimestamp=2021-07-13T00:00:16.730Z InvocationID=980d7a75-ade1-40a1-b7c9-06ad94eadbfe ErrorCode=500 unlock component f91eb779-1810-4803-a317-7e11486ea226 2021-07-13T00:01:04.816Z [qtp817348612-18] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU Compute" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "f898bf2d-367f-477b-b4bf-b4d03036b0d6" INVARIANT_UUID = "da95a1a0-6361-4d21-848b-530cf494cb9e" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=2292e49c-0404-49f1-98f8-b265e70d3a42 TargetServiceName=null EntryTimestamp=2021-07-13T00:00:16.730Z InvocationID=980d7a75-ade1-40a1-b7c9-06ad94eadbfe ErrorCode=500 During change state, another component f91eb779-1810-4803-a317-7e11486ea226 has been created/updated 2021-07-13T00:01:05.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6c74795-918d-48d4-a99e-0f7ea284bebc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:01:05.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6c74795-918d-48d4-a99e-0f7ea284bebc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@65fb30fc 2021-07-13T00:01:05.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6c74795-918d-48d4-a99e-0f7ea284bebc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:01:05.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6c74795-918d-48d4-a99e-0f7ea284bebc PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:01:06.079Z [qtp817348612-18] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU Compute" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "f898bf2d-367f-477b-b4bf-b4d03036b0d6" INVARIANT_UUID = "da95a1a0-6361-4d21-848b-530cf494cb9e" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR ResponseCode=201 RequestID=null ElapsedTime=49347 PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp StatusCode=COMPLETE TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=2292e49c-0404-49f1-98f8-b265e70d3a42 TargetServiceName=null EntryTimestamp=2021-07-13T00:00:16.730Z InvocationID=980d7a75-ade1-40a1-b7c9-06ad94eadbfe ErrorCode=500 LogTimestamp=2021-07-13T00:01:06.077610Z ResponseDesc=Created Exiting. 2021-07-13T00:01:06.079Z [qtp817348612-18] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 SC="201" 2021-07-13T00:01:06.263Z [qtp817348612-17] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=d90e7a1e-cd83-4cc8-ba57-b26debb8c5b4 InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS RemoteHost=10.242.77.79 ServiceName=v1/catalog/upload/multipart ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:01:06.263512Z InvocationID=cd5270f5-de73-4e0a-941c-3c4a3c04b2d0 LogTimestamp=2021-07-13T00:01:06.263571Z Entering 2021-07-13T00:01:06.263Z [qtp817348612-17] INFO o.o.sdc.be.filters.BeServletFilter RequestID=d90e7a1e-cd83-4cc8-ba57-b26debb8c5b4 InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS userId=jh0003 RemoteHost=10.242.77.79 ServiceName=v1/catalog/upload/multipart ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:01:06.263512Z InvocationID=cd5270f5-de73-4e0a-941c-3c4a3c04b2d0 LogTimestamp=2021-07-13T00:01:06.263571Z localAddr=10.242.98.212 remoteAddr=10.242.77.79 POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 2021-07-13T00:01:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:01:07.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:01:08.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b08d4aa4-e244-4bd1-8b78-62d7cce84d5d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:01:08.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b08d4aa4-e244-4bd1-8b78-62d7cce84d5d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3224fb6d 2021-07-13T00:01:08.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b08d4aa4-e244-4bd1-8b78-62d7cce84d5d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:01:08.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b08d4aa4-e244-4bd1-8b78-62d7cce84d5d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:01:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:01:09.367Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:01:09.367Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:01:09.757Z [qtp817348612-17] INFO o.o.s.b.m.o.impl.GraphLockOperation RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:01:06.264Z InvocationID=539c54c2-59b5-4c20-9495-23cbe9805bef PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp lock resource with id 72b1c692-7a3c-4018-93c9-2233b39c9230 2021-07-13T00:01:12.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64430b12-9fa1-485c-a32f-871503e21e99 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:01:12.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64430b12-9fa1-485c-a32f-871503e21e99 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4c76b76b 2021-07-13T00:01:12.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64430b12-9fa1-485c-a32f-871503e21e99 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:01:12.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64430b12-9fa1-485c-a32f-871503e21e99 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:01:12.161Z [qtp817348612-17] INFO o.o.s.b.c.l.LifecycleBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:01:06.264Z InvocationID=539c54c2-59b5-4c20-9495-23cbe9805bef PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp owner of resource 72b1c692-7a3c-4018-93c9-2233b39c9230 is jh0003 2021-07-13T00:01:13.258Z [qtp817348612-17] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VDU VirtualFileStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "add0b207-6eda-4575-aad4-99e5870ce90f" INVARIANT_UUID = "66b605fd-9412-4a5c-9a15-9fdbafa88d6a" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:01:06.264Z InvocationID=539c54c2-59b5-4c20-9495-23cbe9805bef PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:01:13.259Z [qtp817348612-17] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VDU VirtualFileStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "add0b207-6eda-4575-aad4-99e5870ce90f" INVARIANT_UUID = "66b605fd-9412-4a5c-9a15-9fdbafa88d6a" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:01:06.264Z InvocationID=539c54c2-59b5-4c20-9495-23cbe9805bef ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:01:14.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a6acaad-9b59-49f2-8961-da663b1fffd1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:01:14.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a6acaad-9b59-49f2-8961-da663b1fffd1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@520f9483 2021-07-13T00:01:14.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a6acaad-9b59-49f2-8961-da663b1fffd1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:01:14.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a6acaad-9b59-49f2-8961-da663b1fffd1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:01:17.773Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd13d54e-bd55-4597-846f-010106dfaffd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:01:17.773Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd13d54e-bd55-4597-846f-010106dfaffd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6e4f32ee 2021-07-13T00:01:17.773Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd13d54e-bd55-4597-846f-010106dfaffd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:01:17.773Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd13d54e-bd55-4597-846f-010106dfaffd PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:01:20.063Z [qtp817348612-17] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VDU VirtualFileStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "add0b207-6eda-4575-aad4-99e5870ce90f" INVARIANT_UUID = "66b605fd-9412-4a5c-9a15-9fdbafa88d6a" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=344c7fdd-d17b-4664-a357-e3c79a435b90 TargetServiceName=null EntryTimestamp=2021-07-13T00:01:06.264Z InvocationID=539c54c2-59b5-4c20-9495-23cbe9805bef ErrorCode=500 The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:01:20.064Z [qtp817348612-17] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VDU VirtualFileStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "add0b207-6eda-4575-aad4-99e5870ce90f" INVARIANT_UUID = "66b605fd-9412-4a5c-9a15-9fdbafa88d6a" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=344c7fdd-d17b-4664-a357-e3c79a435b90 TargetServiceName=null EntryTimestamp=2021-07-13T00:01:06.264Z InvocationID=539c54c2-59b5-4c20-9495-23cbe9805bef ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:01:21.171Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=560e937e-8cfd-4bcc-ac03-26a24e0af319 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:01:21.171Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=560e937e-8cfd-4bcc-ac03-26a24e0af319 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7b2553d9 2021-07-13T00:01:21.171Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=560e937e-8cfd-4bcc-ac03-26a24e0af319 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:01:21.171Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=560e937e-8cfd-4bcc-ac03-26a24e0af319 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:01:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:01:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:01:23.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d96a8ee3-e345-42d6-8776-df168bb3120c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:01:23.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d96a8ee3-e345-42d6-8776-df168bb3120c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2916e39 2021-07-13T00:01:23.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d96a8ee3-e345-42d6-8776-df168bb3120c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:01:23.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d96a8ee3-e345-42d6-8776-df168bb3120c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:01:24.346Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:01:25.619Z [qtp817348612-17] INFO o.o.s.b.m.o.impl.GraphLockOperation AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VDU VirtualFileStorage" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "add0b207-6eda-4575-aad4-99e5870ce90f" INVARIANT_UUID = "66b605fd-9412-4a5c-9a15-9fdbafa88d6a" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" STATUS = "200" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.Vdu.VirtualFileStorage" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=344c7fdd-d17b-4664-a357-e3c79a435b90 TargetServiceName=null EntryTimestamp=2021-07-13T00:01:06.264Z InvocationID=539c54c2-59b5-4c20-9495-23cbe9805bef ErrorCode=500 lock resource with id faa0f284-e94b-41fc-b008-deaa2ec5bead 2021-07-13T00:01:26.767Z [qtp817348612-17] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VDU VirtualFileStorage" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "add0b207-6eda-4575-aad4-99e5870ce90f" INVARIANT_UUID = "66b605fd-9412-4a5c-9a15-9fdbafa88d6a" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" STATUS = "200" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.Vdu.VirtualFileStorage" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=344c7fdd-d17b-4664-a357-e3c79a435b90 TargetServiceName=null EntryTimestamp=2021-07-13T00:01:06.264Z InvocationID=539c54c2-59b5-4c20-9495-23cbe9805bef ErrorCode=500 owner of resource faa0f284-e94b-41fc-b008-deaa2ec5bead is jh0003 2021-07-13T00:01:26.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=841b14ed-7534-4501-b4e9-6e7b9394c43e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:01:26.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=841b14ed-7534-4501-b4e9-6e7b9394c43e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2c145964 2021-07-13T00:01:26.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=841b14ed-7534-4501-b4e9-6e7b9394c43e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:01:26.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=841b14ed-7534-4501-b4e9-6e7b9394c43e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:01:30.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d8fd01d-8ec9-49f6-b72d-794901884b2c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:01:30.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d8fd01d-8ec9-49f6-b72d-794901884b2c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3bdbd64a 2021-07-13T00:01:30.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d8fd01d-8ec9-49f6-b72d-794901884b2c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:01:30.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d8fd01d-8ec9-49f6-b72d-794901884b2c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:01:33.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18e543e3-89ce-4f37-bbea-f119a46235e6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:01:33.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18e543e3-89ce-4f37-bbea-f119a46235e6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6ab1e173 2021-07-13T00:01:33.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18e543e3-89ce-4f37-bbea-f119a46235e6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:01:33.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18e543e3-89ce-4f37-bbea-f119a46235e6 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:01:35.859Z [qtp817348612-17] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VDU VirtualFileStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "344c7fdd-d17b-4664-a357-e3c79a435b90" INVARIANT_UUID = "66b605fd-9412-4a5c-9a15-9fdbafa88d6a" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=344c7fdd-d17b-4664-a357-e3c79a435b90 TargetServiceName=null EntryTimestamp=2021-07-13T00:01:06.264Z InvocationID=539c54c2-59b5-4c20-9495-23cbe9805bef ErrorCode=500 owner of resource faa0f284-e94b-41fc-b008-deaa2ec5bead is jh0003 2021-07-13T00:01:35.859Z [qtp817348612-17] INFO o.o.s.b.c.l.CertificationChangeTransition AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VDU VirtualFileStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "344c7fdd-d17b-4664-a357-e3c79a435b90" INVARIANT_UUID = "66b605fd-9412-4a5c-9a15-9fdbafa88d6a" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=344c7fdd-d17b-4664-a357-e3c79a435b90 TargetServiceName=null EntryTimestamp=2021-07-13T00:01:06.264Z InvocationID=539c54c2-59b5-4c20-9495-23cbe9805bef ErrorCode=500 validate before certification change. resource name=VDU VirtualFileStorage, oldState=NOT_CERTIFIED_CHECKIN, owner userId=jh0003 2021-07-13T00:01:35.859Z [qtp817348612-17] INFO o.o.s.b.c.l.CertificationChangeTransition AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VDU VirtualFileStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "344c7fdd-d17b-4664-a357-e3c79a435b90" INVARIANT_UUID = "66b605fd-9412-4a5c-9a15-9fdbafa88d6a" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=344c7fdd-d17b-4664-a357-e3c79a435b90 TargetServiceName=null EntryTimestamp=2021-07-13T00:01:06.264Z InvocationID=539c54c2-59b5-4c20-9495-23cbe9805bef ErrorCode=500 start performing certification change for resource faa0f284-e94b-41fc-b008-deaa2ec5bead 2021-07-13T00:01:36.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1951d70-1208-490b-9908-672e8718b180 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:01:36.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1951d70-1208-490b-9908-672e8718b180 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6358aaba 2021-07-13T00:01:36.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1951d70-1208-490b-9908-672e8718b180 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:01:36.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1951d70-1208-490b-9908-672e8718b180 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:01:37.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:01:37.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:01:38.676Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db3f1d82-05fc-4ab5-9c6f-386a9cc8c754 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:01:38.676Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db3f1d82-05fc-4ab5-9c6f-386a9cc8c754 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6d8900cc 2021-07-13T00:01:38.676Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db3f1d82-05fc-4ab5-9c6f-386a9cc8c754 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:01:38.676Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db3f1d82-05fc-4ab5-9c6f-386a9cc8c754 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:01:41.775Z [qtp817348612-17] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VDU VirtualFileStorage" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "344c7fdd-d17b-4664-a357-e3c79a435b90" INVARIANT_UUID = "66b605fd-9412-4a5c-9a15-9fdbafa88d6a" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "72b1c692-7a3c-4018-93c9-2233b39c9230.assettoscatemplate" CURR_ARTIFACT_UUID = "72b1c692-7a3c-4018-93c9-2233b39c9230.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VduVirtualfilestorage-template.yml,0,72b1c692-7a3c-4018-93c9-2233b39c9230.assettoscatemplate,1,ff4bc71b-3788-4d51-bccb-881e50fa2ffb" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=1920b85f-2dc3-4c9e-9f02-9c669a9eec95 TargetServiceName=null EntryTimestamp=2021-07-13T00:01:06.264Z InvocationID=539c54c2-59b5-4c20-9495-23cbe9805bef ErrorCode=500 Starting to create Csar for component VDU VirtualFileStorage 2021-07-13T00:01:41.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32050b1f-6c51-4d8d-b49a-241c518f109e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:01:41.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32050b1f-6c51-4d8d-b49a-241c518f109e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@885e2b1 2021-07-13T00:01:41.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32050b1f-6c51-4d8d-b49a-241c518f109e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:01:41.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32050b1f-6c51-4d8d-b49a-241c518f109e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:01:42.300Z [qtp817348612-17] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VDU VirtualFileStorage" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "344c7fdd-d17b-4664-a357-e3c79a435b90" INVARIANT_UUID = "66b605fd-9412-4a5c-9a15-9fdbafa88d6a" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "72b1c692-7a3c-4018-93c9-2233b39c9230.assettoscatemplate" CURR_ARTIFACT_UUID = "72b1c692-7a3c-4018-93c9-2233b39c9230.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VduVirtualfilestorage-template.yml,0,72b1c692-7a3c-4018-93c9-2233b39c9230.assettoscatemplate,1,ff4bc71b-3788-4d51-bccb-881e50fa2ffb" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=1920b85f-2dc3-4c9e-9f02-9c669a9eec95 TargetServiceName=null EntryTimestamp=2021-07-13T00:01:06.264Z InvocationID=539c54c2-59b5-4c20-9495-23cbe9805bef ErrorCode=500 Ended create Csar for component VDU VirtualFileStorage 2021-07-13T00:01:44.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e8c374c3-77bc-4ac6-abd3-7a2760663ba6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:01:44.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e8c374c3-77bc-4ac6-abd3-7a2760663ba6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6dda20ab 2021-07-13T00:01:44.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e8c374c3-77bc-4ac6-abd3-7a2760663ba6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:01:44.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e8c374c3-77bc-4ac6-abd3-7a2760663ba6 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:01:48.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ccc5bc7c-a129-42c2-9cf8-602693ba2b81 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:01:48.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ccc5bc7c-a129-42c2-9cf8-602693ba2b81 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4b714ad0 2021-07-13T00:01:48.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ccc5bc7c-a129-42c2-9cf8-602693ba2b81 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:01:48.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ccc5bc7c-a129-42c2-9cf8-602693ba2b81 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:01:49.459Z [qtp817348612-17] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU VirtualFileStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "344c7fdd-d17b-4664-a357-e3c79a435b90" INVARIANT_UUID = "66b605fd-9412-4a5c-9a15-9fdbafa88d6a" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=ee7a2eaa-59a6-47d5-a012-59f73f6f593b TargetServiceName=null EntryTimestamp=2021-07-13T00:01:06.264Z InvocationID=539c54c2-59b5-4c20-9495-23cbe9805bef ErrorCode=500 The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:01:49.460Z [qtp817348612-17] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU VirtualFileStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "344c7fdd-d17b-4664-a357-e3c79a435b90" INVARIANT_UUID = "66b605fd-9412-4a5c-9a15-9fdbafa88d6a" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=ee7a2eaa-59a6-47d5-a012-59f73f6f593b TargetServiceName=null EntryTimestamp=2021-07-13T00:01:06.264Z InvocationID=539c54c2-59b5-4c20-9495-23cbe9805bef ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:01:49.460Z [qtp817348612-17] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU VirtualFileStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "344c7fdd-d17b-4664-a357-e3c79a435b90" INVARIANT_UUID = "66b605fd-9412-4a5c-9a15-9fdbafa88d6a" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=ee7a2eaa-59a6-47d5-a012-59f73f6f593b TargetServiceName=null EntryTimestamp=2021-07-13T00:01:06.264Z InvocationID=539c54c2-59b5-4c20-9495-23cbe9805bef ErrorCode=500 unlock component faa0f284-e94b-41fc-b008-deaa2ec5bead 2021-07-13T00:01:49.460Z [qtp817348612-17] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU VirtualFileStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "344c7fdd-d17b-4664-a357-e3c79a435b90" INVARIANT_UUID = "66b605fd-9412-4a5c-9a15-9fdbafa88d6a" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=ee7a2eaa-59a6-47d5-a012-59f73f6f593b TargetServiceName=null EntryTimestamp=2021-07-13T00:01:06.264Z InvocationID=539c54c2-59b5-4c20-9495-23cbe9805bef ErrorCode=500 During change state, another component faa0f284-e94b-41fc-b008-deaa2ec5bead has been created/updated 2021-07-13T00:01:50.475Z [qtp817348612-17] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU VirtualFileStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "344c7fdd-d17b-4664-a357-e3c79a435b90" INVARIANT_UUID = "66b605fd-9412-4a5c-9a15-9fdbafa88d6a" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR ResponseCode=201 RequestID=null ElapsedTime=44211 PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp StatusCode=COMPLETE TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=ee7a2eaa-59a6-47d5-a012-59f73f6f593b TargetServiceName=null EntryTimestamp=2021-07-13T00:01:06.264Z InvocationID=539c54c2-59b5-4c20-9495-23cbe9805bef ErrorCode=500 LogTimestamp=2021-07-13T00:01:50.475353Z ResponseDesc=Created Exiting. 2021-07-13T00:01:50.476Z [qtp817348612-17] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 SC="201" 2021-07-13T00:01:50.664Z [qtp817348612-101] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=21fea3a8-9eea-4e1d-bcf4-4f4b1172e18f InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS RemoteHost=10.242.77.79 ServerIPAddress=10.242.98.212 ServiceName=v1/catalog/upload/multipart ServiceInstanceID=null EntryTimestamp=2021-07-13T00:01:50.664471Z InvocationID=2eb363a7-f84d-4b81-8468-825180663234 LogTimestamp=2021-07-13T00:01:50.664528Z Entering 2021-07-13T00:01:50.665Z [qtp817348612-101] INFO o.o.sdc.be.filters.BeServletFilter RequestID=21fea3a8-9eea-4e1d-bcf4-4f4b1172e18f InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS userId=jh0003 RemoteHost=10.242.77.79 ServerIPAddress=10.242.98.212 ServiceName=v1/catalog/upload/multipart ServiceInstanceID=null EntryTimestamp=2021-07-13T00:01:50.664471Z InvocationID=2eb363a7-f84d-4b81-8468-825180663234 LogTimestamp=2021-07-13T00:01:50.664528Z localAddr=10.242.98.212 remoteAddr=10.242.77.79 POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 2021-07-13T00:01:50.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6137ce61-d0c8-46a1-83f6-8f6fa21102cf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:01:50.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6137ce61-d0c8-46a1-83f6-8f6fa21102cf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6edbac12 2021-07-13T00:01:50.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6137ce61-d0c8-46a1-83f6-8f6fa21102cf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:01:50.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6137ce61-d0c8-46a1-83f6-8f6fa21102cf PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:01:52.611Z [qtp817348612-101] INFO o.o.s.b.m.o.impl.GraphLockOperation RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:01:50.665Z InvocationID=be49d4e7-d5a6-4f98-9252-42ebd113c5b5 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp lock resource with id 93ee3670-f9b9-4914-8f4f-125c6114beba 2021-07-13T00:01:52.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:01:52.925Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:01:53.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=297f3f6a-9d1d-4327-9d85-201f29e95a79 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:01:53.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=297f3f6a-9d1d-4327-9d85-201f29e95a79 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@496c5b12 2021-07-13T00:01:53.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=297f3f6a-9d1d-4327-9d85-201f29e95a79 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:01:53.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=297f3f6a-9d1d-4327-9d85-201f29e95a79 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:01:55.811Z [qtp817348612-101] INFO o.o.s.b.c.l.LifecycleBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:01:50.665Z InvocationID=be49d4e7-d5a6-4f98-9252-42ebd113c5b5 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp owner of resource 93ee3670-f9b9-4914-8f4f-125c6114beba is jh0003 2021-07-13T00:01:56.571Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b13f48ef-9b80-4628-bc1a-11535af18df9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:01:56.571Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b13f48ef-9b80-4628-bc1a-11535af18df9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@758ff5ec 2021-07-13T00:01:56.571Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b13f48ef-9b80-4628-bc1a-11535af18df9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:01:56.571Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b13f48ef-9b80-4628-bc1a-11535af18df9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:01:56.757Z [qtp817348612-101] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VNF External CP" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "9eae1102-d808-4540-b753-70caa72a6b52" INVARIANT_UUID = "185b3f72-2180-4e14-b911-841daa491c4a" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:01:50.665Z InvocationID=be49d4e7-d5a6-4f98-9252-42ebd113c5b5 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:01:56.757Z [qtp817348612-101] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VNF External CP" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "9eae1102-d808-4540-b753-70caa72a6b52" INVARIANT_UUID = "185b3f72-2180-4e14-b911-841daa491c4a" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:01:50.665Z InvocationID=be49d4e7-d5a6-4f98-9252-42ebd113c5b5 ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:01:59.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9dd4f82e-9206-4a84-a890-d01715c62290 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:01:59.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9dd4f82e-9206-4a84-a890-d01715c62290 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@34691647 2021-07-13T00:01:59.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9dd4f82e-9206-4a84-a890-d01715c62290 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:01:59.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9dd4f82e-9206-4a84-a890-d01715c62290 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:02:02.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11887ff6-e686-41a5-b642-2c8cf6571fa1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:02:02.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11887ff6-e686-41a5-b642-2c8cf6571fa1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@76187e4 2021-07-13T00:02:02.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11887ff6-e686-41a5-b642-2c8cf6571fa1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:02:02.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11887ff6-e686-41a5-b642-2c8cf6571fa1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:02:03.770Z [qtp817348612-101] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VNF External CP" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "9eae1102-d808-4540-b753-70caa72a6b52" INVARIANT_UUID = "185b3f72-2180-4e14-b911-841daa491c4a" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=2f978935-faf1-4185-8dab-157c8f2faa0c TargetServiceName=null EntryTimestamp=2021-07-13T00:01:50.665Z InvocationID=be49d4e7-d5a6-4f98-9252-42ebd113c5b5 ErrorCode=500 The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:02:03.771Z [qtp817348612-101] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VNF External CP" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "9eae1102-d808-4540-b753-70caa72a6b52" INVARIANT_UUID = "185b3f72-2180-4e14-b911-841daa491c4a" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=2f978935-faf1-4185-8dab-157c8f2faa0c TargetServiceName=null EntryTimestamp=2021-07-13T00:01:50.665Z InvocationID=be49d4e7-d5a6-4f98-9252-42ebd113c5b5 ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:02:06.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=82967133-32dd-4594-bb9e-6373e445e885 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:02:06.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=82967133-32dd-4594-bb9e-6373e445e885 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2cfef8d1 2021-07-13T00:02:06.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=82967133-32dd-4594-bb9e-6373e445e885 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:02:06.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=82967133-32dd-4594-bb9e-6373e445e885 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:02:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:02:07.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:02:08.760Z [qtp817348612-101] INFO o.o.s.b.m.o.impl.GraphLockOperation AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VNF External CP" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "9eae1102-d808-4540-b753-70caa72a6b52" INVARIANT_UUID = "185b3f72-2180-4e14-b911-841daa491c4a" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" STATUS = "200" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.VnfExtCp" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=2f978935-faf1-4185-8dab-157c8f2faa0c TargetServiceName=null EntryTimestamp=2021-07-13T00:01:50.665Z InvocationID=be49d4e7-d5a6-4f98-9252-42ebd113c5b5 ErrorCode=500 lock resource with id 3bab599d-6118-4f49-b092-e2b0ecffd216 2021-07-13T00:02:08.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=761832a5-4c79-43c5-8f89-b9888a28d6f5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:02:08.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=761832a5-4c79-43c5-8f89-b9888a28d6f5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@26d44837 2021-07-13T00:02:08.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=761832a5-4c79-43c5-8f89-b9888a28d6f5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:02:08.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=761832a5-4c79-43c5-8f89-b9888a28d6f5 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:02:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:02:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:02:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:02:09.860Z [qtp817348612-101] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VNF External CP" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "9eae1102-d808-4540-b753-70caa72a6b52" INVARIANT_UUID = "185b3f72-2180-4e14-b911-841daa491c4a" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" STATUS = "200" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.VnfExtCp" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=2f978935-faf1-4185-8dab-157c8f2faa0c TargetServiceName=null EntryTimestamp=2021-07-13T00:01:50.665Z InvocationID=be49d4e7-d5a6-4f98-9252-42ebd113c5b5 ErrorCode=500 owner of resource 3bab599d-6118-4f49-b092-e2b0ecffd216 is jh0003 2021-07-13T00:02:12.361Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e418ca8-8956-428a-978b-a95b61e595eb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:02:12.361Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e418ca8-8956-428a-978b-a95b61e595eb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5af3b984 2021-07-13T00:02:12.361Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e418ca8-8956-428a-978b-a95b61e595eb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:02:12.361Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e418ca8-8956-428a-978b-a95b61e595eb PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:02:14.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3d8be12-3551-4ea2-a8ea-32f9a6584e61 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:02:14.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3d8be12-3551-4ea2-a8ea-32f9a6584e61 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@51a65b99 2021-07-13T00:02:14.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3d8be12-3551-4ea2-a8ea-32f9a6584e61 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:02:14.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3d8be12-3551-4ea2-a8ea-32f9a6584e61 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:02:18.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d7acd5c-6e64-4abb-baf1-10993a496561 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:02:18.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d7acd5c-6e64-4abb-baf1-10993a496561 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@a45090c 2021-07-13T00:02:18.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d7acd5c-6e64-4abb-baf1-10993a496561 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:02:18.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d7acd5c-6e64-4abb-baf1-10993a496561 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:02:18.560Z [qtp817348612-101] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VNF External CP" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "2f978935-faf1-4185-8dab-157c8f2faa0c" INVARIANT_UUID = "185b3f72-2180-4e14-b911-841daa491c4a" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=2f978935-faf1-4185-8dab-157c8f2faa0c TargetServiceName=null EntryTimestamp=2021-07-13T00:01:50.665Z InvocationID=be49d4e7-d5a6-4f98-9252-42ebd113c5b5 ErrorCode=500 owner of resource 3bab599d-6118-4f49-b092-e2b0ecffd216 is jh0003 2021-07-13T00:02:18.560Z [qtp817348612-101] INFO o.o.s.b.c.l.CertificationChangeTransition AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VNF External CP" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "2f978935-faf1-4185-8dab-157c8f2faa0c" INVARIANT_UUID = "185b3f72-2180-4e14-b911-841daa491c4a" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=2f978935-faf1-4185-8dab-157c8f2faa0c TargetServiceName=null EntryTimestamp=2021-07-13T00:01:50.665Z InvocationID=be49d4e7-d5a6-4f98-9252-42ebd113c5b5 ErrorCode=500 validate before certification change. resource name=VNF External CP, oldState=NOT_CERTIFIED_CHECKIN, owner userId=jh0003 2021-07-13T00:02:18.560Z [qtp817348612-101] INFO o.o.s.b.c.l.CertificationChangeTransition AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VNF External CP" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "2f978935-faf1-4185-8dab-157c8f2faa0c" INVARIANT_UUID = "185b3f72-2180-4e14-b911-841daa491c4a" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=2f978935-faf1-4185-8dab-157c8f2faa0c TargetServiceName=null EntryTimestamp=2021-07-13T00:01:50.665Z InvocationID=be49d4e7-d5a6-4f98-9252-42ebd113c5b5 ErrorCode=500 start performing certification change for resource 3bab599d-6118-4f49-b092-e2b0ecffd216 2021-07-13T00:02:21.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74230be9-ad40-42fc-a083-99ed997d4005 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:02:21.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74230be9-ad40-42fc-a083-99ed997d4005 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e546d83 2021-07-13T00:02:21.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74230be9-ad40-42fc-a083-99ed997d4005 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:02:21.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74230be9-ad40-42fc-a083-99ed997d4005 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:02:22.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:02:22.918Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:02:23.774Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=157af1ad-1dab-46ce-898d-4148a9beb0fd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:02:23.774Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=157af1ad-1dab-46ce-898d-4148a9beb0fd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@492cda54 2021-07-13T00:02:23.774Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=157af1ad-1dab-46ce-898d-4148a9beb0fd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:02:23.774Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=157af1ad-1dab-46ce-898d-4148a9beb0fd PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:02:24.322Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:02:26.071Z [qtp817348612-101] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VNF External CP" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "2f978935-faf1-4185-8dab-157c8f2faa0c" INVARIANT_UUID = "185b3f72-2180-4e14-b911-841daa491c4a" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "93ee3670-f9b9-4914-8f4f-125c6114beba.assettoscatemplate" CURR_ARTIFACT_UUID = "93ee3670-f9b9-4914-8f4f-125c6114beba.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VnfExternalCp-template.yml,0,93ee3670-f9b9-4914-8f4f-125c6114beba.assettoscatemplate,1,a856040e-7ee2-4b8d-adf4-82a54d2156cc" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=b775965a-716e-4bca-8dbc-d3c3677c36f3 TargetServiceName=null EntryTimestamp=2021-07-13T00:01:50.665Z InvocationID=be49d4e7-d5a6-4f98-9252-42ebd113c5b5 ErrorCode=500 Starting to create Csar for component VNF External CP 2021-07-13T00:02:26.075Z [qtp817348612-101] WARN o.o.s.e.AbstractSdncException AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VNF External CP" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "2f978935-faf1-4185-8dab-157c8f2faa0c" INVARIANT_UUID = "185b3f72-2180-4e14-b911-841daa491c4a" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "93ee3670-f9b9-4914-8f4f-125c6114beba.assettoscatemplate" CURR_ARTIFACT_UUID = "93ee3670-f9b9-4914-8f4f-125c6114beba.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VnfExternalCp-template.yml,0,93ee3670-f9b9-4914-8f4f-125c6114beba.assettoscatemplate,1,a856040e-7ee2-4b8d-adf4-82a54d2156cc" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity= ErrorCategory=WARN ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=b775965a-716e-4bca-8dbc-d3c3677c36f3 TargetServiceName=null EntryTimestamp=2021-07-13T00:01:50.665Z InvocationID=be49d4e7-d5a6-4f98-9252-42ebd113c5b5 ErrorCode=900 Received less parameters than expected for error with messageId SVC4063, expected: 1, actual: 0. Missing parameters are padded with null values. 2021-07-13T00:02:26.076Z [qtp817348612-101] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VNF External CP" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "2f978935-faf1-4185-8dab-157c8f2faa0c" INVARIANT_UUID = "185b3f72-2180-4e14-b911-841daa491c4a" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "93ee3670-f9b9-4914-8f4f-125c6114beba.assettoscatemplate" CURR_ARTIFACT_UUID = "93ee3670-f9b9-4914-8f4f-125c6114beba.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VnfExternalCp-template.yml,0,93ee3670-f9b9-4914-8f4f-125c6114beba.assettoscatemplate,1,a856040e-7ee2-4b8d-adf4-82a54d2156cc" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Change LifecycleState ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=b775965a-716e-4bca-8dbc-d3c3677c36f3 TargetServiceName=null EntryTimestamp=2021-07-13T00:01:50.665Z InvocationID=be49d4e7-d5a6-4f98-9252-42ebd113c5b5 ErrorCode=500 Operation towards database failed. 2021-07-13T00:02:26.077Z [qtp817348612-101] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VNF External CP" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "2f978935-faf1-4185-8dab-157c8f2faa0c" INVARIANT_UUID = "185b3f72-2180-4e14-b911-841daa491c4a" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "93ee3670-f9b9-4914-8f4f-125c6114beba.assettoscatemplate" CURR_ARTIFACT_UUID = "93ee3670-f9b9-4914-8f4f-125c6114beba.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VnfExternalCp-template.yml,0,93ee3670-f9b9-4914-8f4f-125c6114beba.assettoscatemplate,1,a856040e-7ee2-4b8d-adf4-82a54d2156cc" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Change LifecycleState ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=b775965a-716e-4bca-8dbc-d3c3677c36f3 TargetServiceName=null EntryTimestamp=2021-07-13T00:01:50.665Z InvocationID=be49d4e7-d5a6-4f98-9252-42ebd113c5b5 ErrorCode=500 unlock component 3bab599d-6118-4f49-b092-e2b0ecffd216 2021-07-13T00:02:26.077Z [qtp817348612-101] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VNF External CP" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "2f978935-faf1-4185-8dab-157c8f2faa0c" INVARIANT_UUID = "185b3f72-2180-4e14-b911-841daa491c4a" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "93ee3670-f9b9-4914-8f4f-125c6114beba.assettoscatemplate" CURR_ARTIFACT_UUID = "93ee3670-f9b9-4914-8f4f-125c6114beba.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VnfExternalCp-template.yml,0,93ee3670-f9b9-4914-8f4f-125c6114beba.assettoscatemplate,1,a856040e-7ee2-4b8d-adf4-82a54d2156cc" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Change LifecycleState ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=b775965a-716e-4bca-8dbc-d3c3677c36f3 TargetServiceName=null EntryTimestamp=2021-07-13T00:01:50.665Z InvocationID=be49d4e7-d5a6-4f98-9252-42ebd113c5b5 ErrorCode=500 During change state, another component 3bab599d-6118-4f49-b092-e2b0ecffd216 has been created/updated 2021-07-13T00:02:27.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f5cda59-5570-4e61-826c-33f8476a87df PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:02:27.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f5cda59-5570-4e61-826c-33f8476a87df PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ab52a7b 2021-07-13T00:02:27.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f5cda59-5570-4e61-826c-33f8476a87df PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:02:27.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f5cda59-5570-4e61-826c-33f8476a87df PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:02:27.814Z [qtp817348612-101] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VNF External CP" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "2f978935-faf1-4185-8dab-157c8f2faa0c" INVARIANT_UUID = "185b3f72-2180-4e14-b911-841daa491c4a" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "93ee3670-f9b9-4914-8f4f-125c6114beba.assettoscatemplate" CURR_ARTIFACT_UUID = "93ee3670-f9b9-4914-8f4f-125c6114beba.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VnfExternalCp-template.yml,0,93ee3670-f9b9-4914-8f4f-125c6114beba.assettoscatemplate,1,a856040e-7ee2-4b8d-adf4-82a54d2156cc" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Change LifecycleState - Certify ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=b775965a-716e-4bca-8dbc-d3c3677c36f3 TargetServiceName=null EntryTimestamp=2021-07-13T00:01:50.665Z InvocationID=be49d4e7-d5a6-4f98-9252-42ebd113c5b5 ErrorCode=500 Unexpected error during operation 2021-07-13T00:02:27.814Z [qtp817348612-101] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VNF External CP" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "2f978935-faf1-4185-8dab-157c8f2faa0c" INVARIANT_UUID = "185b3f72-2180-4e14-b911-841daa491c4a" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "93ee3670-f9b9-4914-8f4f-125c6114beba.assettoscatemplate" CURR_ARTIFACT_UUID = "93ee3670-f9b9-4914-8f4f-125c6114beba.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VnfExternalCp-template.yml,0,93ee3670-f9b9-4914-8f4f-125c6114beba.assettoscatemplate,1,a856040e-7ee2-4b8d-adf4-82a54d2156cc" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Import Resource vnfExtCp.yml ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=b775965a-716e-4bca-8dbc-d3c3677c36f3 TargetServiceName=null EntryTimestamp=2021-07-13T00:01:50.665Z InvocationID=be49d4e7-d5a6-4f98-9252-42ebd113c5b5 ErrorCode=500 Unexpected error during operation 2021-07-13T00:02:27.819Z [qtp817348612-101] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VNF External CP" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "" INVARIANT_UUID = "" PREV_VERSION = "1.0" CURR_VERSION = "" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "" STATUS = "404" DESC = "SVC4063: Error: Requested 'null' resource was not found." TOSCA_NODE_TYPE = "" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Upload Resource ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=b775965a-716e-4bca-8dbc-d3c3677c36f3 TargetServiceName=null EntryTimestamp=2021-07-13T00:01:50.665Z InvocationID=be49d4e7-d5a6-4f98-9252-42ebd113c5b5 ErrorCode=900 Unexpected error during BE REST API execution 2021-07-13T00:02:27.821Z [qtp817348612-101] ERROR o.o.s.b.s.ResourceUploadServlet AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VNF External CP" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "" INVARIANT_UUID = "" PREV_VERSION = "1.0" CURR_VERSION = "" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "" STATUS = "404" DESC = "SVC4063: Error: Requested 'null' resource was not found." TOSCA_NODE_TYPE = "" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Upload Resource ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=b775965a-716e-4bca-8dbc-d3c3677c36f3 TargetServiceName=null EntryTimestamp=2021-07-13T00:01:50.665Z InvocationID=be49d4e7-d5a6-4f98-9252-42ebd113c5b5 ErrorCode=900 #uploadMultipart('multipart', /tmp/rep14770857855532413874tmp, 'form-data; filename="vnfExtCp.zip"; name="resourceZip"', '{"payloadName": "vnfExtCp.yml", "contactId": "jh0003", "name": "VNF External CP", "description": "VNF External CP", "resourceIconPath": "network", "resourceType": "CP", "categories": [{"name": "Generic", "subcategories": [{"name": "Infrastructure"}]}], "vendorName": "ETSI", "vendorRelease": "2.7.1", "tags": ["VNF External CP"]}', 'Request(POST https://sdc-be.onap:8443/sdc2/rest/v1/catalog/upload/multipart?createNewVersion=true)@62228854', 'jh0003', true): thrown org.openecomp.sdc.be.components.impl.exceptions.ByResponseFormatComponentException out of org.openecomp.sdc.be.components.impl.ArtifactsBusinessLogic#lambda$decodeToscaArtifactPayload$5[371] in 37s 2021-07-13T00:02:27.823Z [qtp817348612-101] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VNF External CP" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "" INVARIANT_UUID = "" PREV_VERSION = "1.0" CURR_VERSION = "" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "" STATUS = "404" DESC = "SVC4063: Error: Requested 'null' resource was not found." TOSCA_NODE_TYPE = "" SupportablityAction=GENERATE CSAR ResponseCode=404 RequestID=null ElapsedTime=37158 PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp StatusCode=ERROR TargetEntity=Upload Resource ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=b775965a-716e-4bca-8dbc-d3c3677c36f3 TargetServiceName=null ErrorDesc=Not Found EntryTimestamp=2021-07-13T00:01:50.665Z InvocationID=be49d4e7-d5a6-4f98-9252-42ebd113c5b5 ErrorCode=404 LogTimestamp=2021-07-13T00:02:27.823369Z ResponseDesc=Not Found Exiting. 2021-07-13T00:02:27.824Z [qtp817348612-101] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 SC="404" 2021-07-13T00:02:29.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba203c35-f7b8-41b3-86f4-bdd5071e5e26 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:02:29.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba203c35-f7b8-41b3-86f4-bdd5071e5e26 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@89a4b48 2021-07-13T00:02:29.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba203c35-f7b8-41b3-86f4-bdd5071e5e26 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:02:29.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba203c35-f7b8-41b3-86f4-bdd5071e5e26 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:02:32.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=efc6d94b-6ef0-4d3f-94c3-1ffc9081f317 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:02:32.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=efc6d94b-6ef0-4d3f-94c3-1ffc9081f317 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@258f07ad 2021-07-13T00:02:32.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=efc6d94b-6ef0-4d3f-94c3-1ffc9081f317 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:02:32.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=efc6d94b-6ef0-4d3f-94c3-1ffc9081f317 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:02:35.562Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3eb0eac0-865b-44ce-abe1-8fac4b077d43 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:02:35.562Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3eb0eac0-865b-44ce-abe1-8fac4b077d43 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@ade8732 2021-07-13T00:02:35.562Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3eb0eac0-865b-44ce-abe1-8fac4b077d43 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:02:35.562Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3eb0eac0-865b-44ce-abe1-8fac4b077d43 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:02:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:02:37.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:02:38.119Z [qtp817348612-96] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=fa3c6888-5a73-49ed-89df-1ac5437ec4ed InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS RemoteHost=10.242.77.79 ServerIPAddress=10.242.98.212 ServiceName=v1/catalog/upload/multipart ServiceInstanceID=null EntryTimestamp=2021-07-13T00:02:38.118718Z InvocationID=b13520cf-0c48-4661-8e38-883c5115d3e8 LogTimestamp=2021-07-13T00:02:38.118833Z Entering 2021-07-13T00:02:38.119Z [qtp817348612-96] INFO o.o.sdc.be.filters.BeServletFilter RequestID=fa3c6888-5a73-49ed-89df-1ac5437ec4ed InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS userId=jh0003 RemoteHost=10.242.77.79 ServerIPAddress=10.242.98.212 ServiceName=v1/catalog/upload/multipart ServiceInstanceID=null EntryTimestamp=2021-07-13T00:02:38.118718Z InvocationID=b13520cf-0c48-4661-8e38-883c5115d3e8 LogTimestamp=2021-07-13T00:02:38.118833Z localAddr=10.242.98.212 remoteAddr=10.242.77.79 POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 2021-07-13T00:02:38.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a5fd7e5-edd9-4183-a592-a2fbe773cb5e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:02:38.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a5fd7e5-edd9-4183-a592-a2fbe773cb5e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@9bb82e0 2021-07-13T00:02:38.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a5fd7e5-edd9-4183-a592-a2fbe773cb5e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:02:38.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a5fd7e5-edd9-4183-a592-a2fbe773cb5e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:02:41.360Z [qtp817348612-96] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Import Resource vnfExtCp.yml ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:02:38.119Z InvocationID=d019023d-8334-495a-9ab8-703478b00a74 ErrorCode=500 Unexpected error during operation 2021-07-13T00:02:41.366Z [qtp817348612-96] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VNF External CP" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "" INVARIANT_UUID = "" PREV_VERSION = "1.0" CURR_VERSION = "" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "" STATUS = "409" DESC = "SVC4083: Error: Version of this VNF External CP was already promoted." TOSCA_NODE_TYPE = "" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Upload Resource ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:02:38.119Z InvocationID=d019023d-8334-495a-9ab8-703478b00a74 ErrorCode=900 Unexpected error during BE REST API execution 2021-07-13T00:02:41.368Z [qtp817348612-96] ERROR o.o.s.b.s.ResourceUploadServlet AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VNF External CP" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "" INVARIANT_UUID = "" PREV_VERSION = "1.0" CURR_VERSION = "" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "" STATUS = "409" DESC = "SVC4083: Error: Version of this VNF External CP was already promoted." TOSCA_NODE_TYPE = "" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Upload Resource ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:02:38.119Z InvocationID=d019023d-8334-495a-9ab8-703478b00a74 ErrorCode=900 #uploadMultipart('multipart', /tmp/rep3653747539237476741tmp, 'form-data; filename="vnfExtCp.zip"; name="resourceZip"', '{"payloadName": "vnfExtCp.yml", "contactId": "jh0003", "name": "VNF External CP", "description": "VNF External CP", "resourceIconPath": "network", "resourceType": "CP", "categories": [{"name": "Generic", "subcategories": [{"name": "Infrastructure"}]}], "vendorName": "ETSI", "vendorRelease": "2.7.1", "tags": ["VNF External CP"]}', 'Request(POST https://sdc-be.onap:8443/sdc2/rest/v1/catalog/upload/multipart?createNewVersion=true)@62228854', 'jh0003', true): thrown org.openecomp.sdc.be.components.impl.exceptions.ByActionStatusComponentException out of org.openecomp.sdc.be.components.impl.ResourceImportManager#importCertifiedResource[184] in 3s 2021-07-13T00:02:41.369Z [qtp817348612-96] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VNF External CP" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "" INVARIANT_UUID = "" PREV_VERSION = "1.0" CURR_VERSION = "" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "" STATUS = "409" DESC = "SVC4083: Error: Version of this VNF External CP was already promoted." TOSCA_NODE_TYPE = "" ResponseCode=409 RequestID=null ElapsedTime=3250 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp StatusCode=ERROR TargetEntity=Upload Resource ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null ErrorDesc=Conflict EntryTimestamp=2021-07-13T00:02:38.119Z InvocationID=d019023d-8334-495a-9ab8-703478b00a74 ErrorCode=409 LogTimestamp=2021-07-13T00:02:41.369185Z ResponseDesc=Conflict Exiting. 2021-07-13T00:02:41.370Z [qtp817348612-96] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 SC="409" 2021-07-13T00:02:41.465Z [qtp817348612-101] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=f0b21211-9e88-40c1-afad-a711f924f29c InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS RemoteHost=10.242.77.79 ServiceName=v1/catalog/upload/multipart ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:02:41.464908Z InvocationID=f75109d3-ef08-49ef-8847-49121473bf8c LogTimestamp=2021-07-13T00:02:41.464964Z Entering 2021-07-13T00:02:41.465Z [qtp817348612-101] INFO o.o.sdc.be.filters.BeServletFilter RequestID=f0b21211-9e88-40c1-afad-a711f924f29c InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS userId=jh0003 RemoteHost=10.242.77.79 ServiceName=v1/catalog/upload/multipart ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:02:41.464908Z InvocationID=f75109d3-ef08-49ef-8847-49121473bf8c LogTimestamp=2021-07-13T00:02:41.464964Z localAddr=10.242.98.212 remoteAddr=10.242.77.79 POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 2021-07-13T00:02:41.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15f3d084-d1e5-4dc2-a74a-9a64317fd0e6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:02:41.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15f3d084-d1e5-4dc2-a74a-9a64317fd0e6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@53a14894 2021-07-13T00:02:41.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15f3d084-d1e5-4dc2-a74a-9a64317fd0e6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:02:41.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15f3d084-d1e5-4dc2-a74a-9a64317fd0e6 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:02:45.057Z [qtp817348612-101] INFO o.o.s.b.m.o.impl.GraphLockOperation RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:02:41.465Z InvocationID=e9d5f1cc-a147-45a6-a62b-1aedf2ae2bf0 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp lock resource with id 0965cd8d-df6d-4a13-aa9f-1f62076c7b4b 2021-07-13T00:02:45.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=921f5cd9-cf21-41be-8815-2335298d90e4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:02:45.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=921f5cd9-cf21-41be-8815-2335298d90e4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e4e8f22 2021-07-13T00:02:45.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=921f5cd9-cf21-41be-8815-2335298d90e4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:02:45.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=921f5cd9-cf21-41be-8815-2335298d90e4 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:02:47.611Z [qtp817348612-101] INFO o.o.s.b.c.l.LifecycleBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:02:41.465Z InvocationID=e9d5f1cc-a147-45a6-a62b-1aedf2ae2bf0 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp owner of resource 0965cd8d-df6d-4a13-aa9f-1f62076c7b4b is jh0003 2021-07-13T00:02:47.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6a9acb9-349f-47c2-a4c2-228bc3e88806 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:02:47.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6a9acb9-349f-47c2-a4c2-228bc3e88806 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@70d76ece 2021-07-13T00:02:47.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6a9acb9-349f-47c2-a4c2-228bc3e88806 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:02:47.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6a9acb9-349f-47c2-a4c2-228bc3e88806 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:02:48.513Z [qtp817348612-101] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "44a1c695-0ea2-4e45-ac74-7807d6a930d6" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:02:41.465Z InvocationID=e9d5f1cc-a147-45a6-a62b-1aedf2ae2bf0 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:02:48.514Z [qtp817348612-101] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "44a1c695-0ea2-4e45-ac74-7807d6a930d6" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:02:41.465Z InvocationID=e9d5f1cc-a147-45a6-a62b-1aedf2ae2bf0 ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:02:50.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=764acf6e-a73b-4356-8c17-8e208d50eecc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:02:50.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=764acf6e-a73b-4356-8c17-8e208d50eecc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@25ae5eb6 2021-07-13T00:02:50.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=764acf6e-a73b-4356-8c17-8e208d50eecc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:02:50.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=764acf6e-a73b-4356-8c17-8e208d50eecc PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:02:52.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:02:52.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:02:54.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2851306b-64fe-4f51-8ef7-33485d558f0c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:02:54.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2851306b-64fe-4f51-8ef7-33485d558f0c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7c92577f 2021-07-13T00:02:54.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2851306b-64fe-4f51-8ef7-33485d558f0c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:02:54.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2851306b-64fe-4f51-8ef7-33485d558f0c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:02:54.967Z [qtp817348612-101] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "44a1c695-0ea2-4e45-ac74-7807d6a930d6" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=8ea87a7d-0be7-493a-8692-d3ce1aaf10ee TargetServiceName=null EntryTimestamp=2021-07-13T00:02:41.465Z InvocationID=e9d5f1cc-a147-45a6-a62b-1aedf2ae2bf0 ErrorCode=500 The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:02:54.968Z [qtp817348612-101] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "44a1c695-0ea2-4e45-ac74-7807d6a930d6" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=8ea87a7d-0be7-493a-8692-d3ce1aaf10ee TargetServiceName=null EntryTimestamp=2021-07-13T00:02:41.465Z InvocationID=e9d5f1cc-a147-45a6-a62b-1aedf2ae2bf0 ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:02:57.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ae2b4af-a13f-4c91-838f-8415f2d0d0bb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:02:57.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ae2b4af-a13f-4c91-838f-8415f2d0d0bb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@65fe050 2021-07-13T00:02:57.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ae2b4af-a13f-4c91-838f-8415f2d0d0bb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:02:57.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ae2b4af-a13f-4c91-838f-8415f2d0d0bb PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:02:59.871Z [qtp817348612-101] INFO o.o.s.b.m.o.impl.GraphLockOperation AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "44a1c695-0ea2-4e45-ac74-7807d6a930d6" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" STATUS = "200" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.VduCp" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=8ea87a7d-0be7-493a-8692-d3ce1aaf10ee TargetServiceName=null EntryTimestamp=2021-07-13T00:02:41.465Z InvocationID=e9d5f1cc-a147-45a6-a62b-1aedf2ae2bf0 ErrorCode=500 lock resource with id 6ad3e8dd-be5e-42f0-8e2d-00c01977d9b7 2021-07-13T00:03:00.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e690369a-fc32-41d0-b98f-1442933a18e7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:03:00.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e690369a-fc32-41d0-b98f-1442933a18e7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4db55f4b 2021-07-13T00:03:00.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e690369a-fc32-41d0-b98f-1442933a18e7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:03:00.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e690369a-fc32-41d0-b98f-1442933a18e7 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:03:00.870Z [qtp817348612-101] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "44a1c695-0ea2-4e45-ac74-7807d6a930d6" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" STATUS = "200" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.VduCp" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=8ea87a7d-0be7-493a-8692-d3ce1aaf10ee TargetServiceName=null EntryTimestamp=2021-07-13T00:02:41.465Z InvocationID=e9d5f1cc-a147-45a6-a62b-1aedf2ae2bf0 ErrorCode=500 owner of resource 6ad3e8dd-be5e-42f0-8e2d-00c01977d9b7 is jh0003 2021-07-13T00:03:03.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25ce1871-182f-4c4e-9adf-a7326ac78ddf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:03:03.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25ce1871-182f-4c4e-9adf-a7326ac78ddf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1ab9e3a5 2021-07-13T00:03:03.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25ce1871-182f-4c4e-9adf-a7326ac78ddf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:03:03.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25ce1871-182f-4c4e-9adf-a7326ac78ddf PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:03:05.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9933fff2-69d6-4dce-9420-f66a795c0eb5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:03:05.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9933fff2-69d6-4dce-9420-f66a795c0eb5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@74fe6622 2021-07-13T00:03:05.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9933fff2-69d6-4dce-9420-f66a795c0eb5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:03:05.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9933fff2-69d6-4dce-9420-f66a795c0eb5 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:03:07.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:03:07.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:03:08.711Z [qtp817348612-101] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "8ea87a7d-0be7-493a-8692-d3ce1aaf10ee" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=8ea87a7d-0be7-493a-8692-d3ce1aaf10ee TargetServiceName=null EntryTimestamp=2021-07-13T00:02:41.465Z InvocationID=e9d5f1cc-a147-45a6-a62b-1aedf2ae2bf0 ErrorCode=500 owner of resource 6ad3e8dd-be5e-42f0-8e2d-00c01977d9b7 is jh0003 2021-07-13T00:03:08.711Z [qtp817348612-101] INFO o.o.s.b.c.l.CertificationChangeTransition AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "8ea87a7d-0be7-493a-8692-d3ce1aaf10ee" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=8ea87a7d-0be7-493a-8692-d3ce1aaf10ee TargetServiceName=null EntryTimestamp=2021-07-13T00:02:41.465Z InvocationID=e9d5f1cc-a147-45a6-a62b-1aedf2ae2bf0 ErrorCode=500 validate before certification change. resource name=VDU Cp, oldState=NOT_CERTIFIED_CHECKIN, owner userId=jh0003 2021-07-13T00:03:08.711Z [qtp817348612-101] INFO o.o.s.b.c.l.CertificationChangeTransition AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "8ea87a7d-0be7-493a-8692-d3ce1aaf10ee" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=8ea87a7d-0be7-493a-8692-d3ce1aaf10ee TargetServiceName=null EntryTimestamp=2021-07-13T00:02:41.465Z InvocationID=e9d5f1cc-a147-45a6-a62b-1aedf2ae2bf0 ErrorCode=500 start performing certification change for resource 6ad3e8dd-be5e-42f0-8e2d-00c01977d9b7 2021-07-13T00:03:09.171Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee6010bd-77c4-4acb-b80f-27f4df89f7a8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:03:09.171Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee6010bd-77c4-4acb-b80f-27f4df89f7a8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@d5a558d 2021-07-13T00:03:09.171Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee6010bd-77c4-4acb-b80f-27f4df89f7a8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:03:09.171Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ee6010bd-77c4-4acb-b80f-27f4df89f7a8 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:03:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:03:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:03:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:03:11.778Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d17f49f-6873-4269-86d1-ca340e1f8cf9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:03:11.778Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d17f49f-6873-4269-86d1-ca340e1f8cf9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6af20c6a 2021-07-13T00:03:11.778Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d17f49f-6873-4269-86d1-ca340e1f8cf9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:03:11.778Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d17f49f-6873-4269-86d1-ca340e1f8cf9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:03:15.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5ee4f48-36c0-499d-a831-b2b8d8e803cb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:03:15.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5ee4f48-36c0-499d-a831-b2b8d8e803cb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3f836e5a 2021-07-13T00:03:15.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5ee4f48-36c0-499d-a831-b2b8d8e803cb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:03:15.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5ee4f48-36c0-499d-a831-b2b8d8e803cb PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:03:16.160Z [qtp817348612-101] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "8ea87a7d-0be7-493a-8692-d3ce1aaf10ee" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "0965cd8d-df6d-4a13-aa9f-1f62076c7b4b.assettoscatemplate" CURR_ARTIFACT_UUID = "0965cd8d-df6d-4a13-aa9f-1f62076c7b4b.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VduCp-template.yml,0,0965cd8d-df6d-4a13-aa9f-1f62076c7b4b.assettoscatemplate,1,8a9d2bc9-f8e6-40ee-a1ea-5409e2d7f775" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=6d00cc50-78d5-416a-b4fe-f9f3f316434a TargetServiceName=null EntryTimestamp=2021-07-13T00:02:41.465Z InvocationID=e9d5f1cc-a147-45a6-a62b-1aedf2ae2bf0 ErrorCode=500 Starting to create Csar for component VDU Cp 2021-07-13T00:03:16.433Z [qtp817348612-101] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "8ea87a7d-0be7-493a-8692-d3ce1aaf10ee" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "0965cd8d-df6d-4a13-aa9f-1f62076c7b4b.assettoscatemplate" CURR_ARTIFACT_UUID = "0965cd8d-df6d-4a13-aa9f-1f62076c7b4b.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VduCp-template.yml,0,0965cd8d-df6d-4a13-aa9f-1f62076c7b4b.assettoscatemplate,1,8a9d2bc9-f8e6-40ee-a1ea-5409e2d7f775" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=6d00cc50-78d5-416a-b4fe-f9f3f316434a TargetServiceName=null EntryTimestamp=2021-07-13T00:02:41.465Z InvocationID=e9d5f1cc-a147-45a6-a62b-1aedf2ae2bf0 ErrorCode=500 Ended create Csar for component VDU Cp 2021-07-13T00:03:18.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42a97423-fb2e-4a87-8a81-13c293f35439 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:03:18.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42a97423-fb2e-4a87-8a81-13c293f35439 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@38cbc0fe 2021-07-13T00:03:18.166Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42a97423-fb2e-4a87-8a81-13c293f35439 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:03:18.166Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42a97423-fb2e-4a87-8a81-13c293f35439 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:03:20.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88c48ada-e8de-488b-ab24-3a9eced4e4d0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:03:20.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88c48ada-e8de-488b-ab24-3a9eced4e4d0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@67620c57 2021-07-13T00:03:20.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88c48ada-e8de-488b-ab24-3a9eced4e4d0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:03:20.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88c48ada-e8de-488b-ab24-3a9eced4e4d0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:03:21.814Z [qtp817348612-101] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "8ea87a7d-0be7-493a-8692-d3ce1aaf10ee" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=8c1d5bc2-9a3c-4a3e-b3b8-ba9324109ece TargetServiceName=null EntryTimestamp=2021-07-13T00:02:41.465Z InvocationID=e9d5f1cc-a147-45a6-a62b-1aedf2ae2bf0 ErrorCode=500 The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:03:21.815Z [qtp817348612-101] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "8ea87a7d-0be7-493a-8692-d3ce1aaf10ee" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=8c1d5bc2-9a3c-4a3e-b3b8-ba9324109ece TargetServiceName=null EntryTimestamp=2021-07-13T00:02:41.465Z InvocationID=e9d5f1cc-a147-45a6-a62b-1aedf2ae2bf0 ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:03:21.815Z [qtp817348612-101] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "8ea87a7d-0be7-493a-8692-d3ce1aaf10ee" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=8c1d5bc2-9a3c-4a3e-b3b8-ba9324109ece TargetServiceName=null EntryTimestamp=2021-07-13T00:02:41.465Z InvocationID=e9d5f1cc-a147-45a6-a62b-1aedf2ae2bf0 ErrorCode=500 unlock component 6ad3e8dd-be5e-42f0-8e2d-00c01977d9b7 2021-07-13T00:03:21.815Z [qtp817348612-101] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "8ea87a7d-0be7-493a-8692-d3ce1aaf10ee" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=8c1d5bc2-9a3c-4a3e-b3b8-ba9324109ece TargetServiceName=null EntryTimestamp=2021-07-13T00:02:41.465Z InvocationID=e9d5f1cc-a147-45a6-a62b-1aedf2ae2bf0 ErrorCode=500 During change state, another component 6ad3e8dd-be5e-42f0-8e2d-00c01977d9b7 has been created/updated 2021-07-13T00:03:22.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:03:22.925Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:03:23.331Z [qtp817348612-101] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "8ea87a7d-0be7-493a-8692-d3ce1aaf10ee" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR ResponseCode=201 RequestID=null ElapsedTime=41865 PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp StatusCode=COMPLETE TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=8c1d5bc2-9a3c-4a3e-b3b8-ba9324109ece TargetServiceName=null EntryTimestamp=2021-07-13T00:02:41.465Z InvocationID=e9d5f1cc-a147-45a6-a62b-1aedf2ae2bf0 ErrorCode=500 LogTimestamp=2021-07-13T00:03:23.330552Z ResponseDesc=Created Exiting. 2021-07-13T00:03:23.332Z [qtp817348612-101] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 SC="201" 2021-07-13T00:03:23.567Z [qtp817348612-17] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=d7c2c97f-0255-4b0f-a653-0d75b9e2e5fd InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS RemoteHost=10.242.77.79 ServiceName=v1/catalog/upload/multipart ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:03:23.567366Z InvocationID=b4a06c6e-f465-436e-bb6e-052e0a93497c LogTimestamp=2021-07-13T00:03:23.567432Z Entering 2021-07-13T00:03:23.567Z [qtp817348612-17] INFO o.o.sdc.be.filters.BeServletFilter RequestID=d7c2c97f-0255-4b0f-a653-0d75b9e2e5fd InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS userId=jh0003 RemoteHost=10.242.77.79 ServiceName=v1/catalog/upload/multipart ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:03:23.567366Z InvocationID=b4a06c6e-f465-436e-bb6e-052e0a93497c LogTimestamp=2021-07-13T00:03:23.567432Z localAddr=10.242.98.212 remoteAddr=10.242.77.79 POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 2021-07-13T00:03:23.669Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2821c557-5a76-4699-bc05-1c02ee3edfe3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:03:23.669Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2821c557-5a76-4699-bc05-1c02ee3edfe3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1d09cd8d 2021-07-13T00:03:23.669Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2821c557-5a76-4699-bc05-1c02ee3edfe3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:03:23.669Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2821c557-5a76-4699-bc05-1c02ee3edfe3 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:03:24.321Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:03:27.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d03f993f-5fb2-45a7-9cb0-c4ee2e35433b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:03:27.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d03f993f-5fb2-45a7-9cb0-c4ee2e35433b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@395c91c8 2021-07-13T00:03:27.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d03f993f-5fb2-45a7-9cb0-c4ee2e35433b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:03:27.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d03f993f-5fb2-45a7-9cb0-c4ee2e35433b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:03:29.661Z [qtp817348612-17] INFO o.o.s.b.c.i.ComponentBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:03:23.568Z InvocationID=f160719b-fbce-4e25-9e48-61da8baacedf PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp LifecycleState cannot be defined by user. This field will be overridden by the application 2021-07-13T00:03:29.661Z [qtp817348612-17] INFO o.o.s.b.c.i.ComponentBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:03:23.568Z InvocationID=f160719b-fbce-4e25-9e48-61da8baacedf PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Version cannot be defined by user. This field will be overridden by the application 2021-07-13T00:03:29.661Z [qtp817348612-17] INFO o.o.s.b.c.i.ComponentBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:03:23.568Z InvocationID=f160719b-fbce-4e25-9e48-61da8baacedf PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Creator cannot be defined by user. This field will be overridden by the application 2021-07-13T00:03:29.661Z [qtp817348612-17] INFO o.o.s.b.c.i.ComponentBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:03:23.568Z InvocationID=f160719b-fbce-4e25-9e48-61da8baacedf PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Is Highest Version cannot be defined by user. This field will be overridden by the application 2021-07-13T00:03:30.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed858e35-68dd-4e16-96dd-39c8f2f5cfea PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:03:30.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed858e35-68dd-4e16-96dd-39c8f2f5cfea PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5ff66a2d 2021-07-13T00:03:30.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed858e35-68dd-4e16-96dd-39c8f2f5cfea PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:03:30.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed858e35-68dd-4e16-96dd-39c8f2f5cfea PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:03:30.458Z [qtp817348612-17] INFO o.o.s.b.m.o.impl.GraphLockOperation RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:03:23.568Z InvocationID=f160719b-fbce-4e25-9e48-61da8baacedf PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp lock resource with name VipCp 2021-07-13T00:03:33.269Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17375b4f-82cc-46a4-9aca-14b67698ae2a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:03:33.269Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17375b4f-82cc-46a4-9aca-14b67698ae2a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2881d530 2021-07-13T00:03:33.269Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17375b4f-82cc-46a4-9aca-14b67698ae2a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:03:33.269Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17375b4f-82cc-46a4-9aca-14b67698ae2a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:03:35.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e70d296b-1c73-4468-b7b5-78fe820e99cc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:03:35.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e70d296b-1c73-4468-b7b5-78fe820e99cc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7b371008 2021-07-13T00:03:35.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e70d296b-1c73-4468-b7b5-78fe820e99cc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:03:35.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e70d296b-1c73-4468-b7b5-78fe820e99cc PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:03:37.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:03:37.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:03:38.062Z [qtp817348612-17] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VIP Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "220232df-ed9d-403d-997b-881d762b462d" INVARIANT_UUID = "c1ac5760-c02c-46e3-b34d-f1cac3149a9e" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "" CURR_STATE = "CERTIFIED" STATUS = "201" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.VipCp" RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=220232df-ed9d-403d-997b-881d762b462d EntryTimestamp=2021-07-13T00:03:23.568Z InvocationID=f160719b-fbce-4e25-9e48-61da8baacedf PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:03:38.063Z [qtp817348612-17] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VIP Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "220232df-ed9d-403d-997b-881d762b462d" INVARIANT_UUID = "c1ac5760-c02c-46e3-b34d-f1cac3149a9e" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "" CURR_STATE = "CERTIFIED" STATUS = "201" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.VipCp" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=220232df-ed9d-403d-997b-881d762b462d TargetServiceName=null EntryTimestamp=2021-07-13T00:03:23.568Z InvocationID=f160719b-fbce-4e25-9e48-61da8baacedf ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:03:39.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0d890bd-e233-45da-9133-74200a39d4c2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:03:39.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0d890bd-e233-45da-9133-74200a39d4c2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@26380c77 2021-07-13T00:03:39.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0d890bd-e233-45da-9133-74200a39d4c2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:03:39.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d0d890bd-e233-45da-9133-74200a39d4c2 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:03:41.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca737e6b-5578-4a0c-8fb1-443427d7bca5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:03:41.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca737e6b-5578-4a0c-8fb1-443427d7bca5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4a73ad9f 2021-07-13T00:03:41.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca737e6b-5578-4a0c-8fb1-443427d7bca5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:03:41.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca737e6b-5578-4a0c-8fb1-443427d7bca5 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:03:42.167Z [qtp817348612-17] INFO o.o.s.b.m.o.impl.GraphLockOperation AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VIP Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "220232df-ed9d-403d-997b-881d762b462d" INVARIANT_UUID = "c1ac5760-c02c-46e3-b34d-f1cac3149a9e" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "" CURR_STATE = "CERTIFIED" STATUS = "201" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.VipCp" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=220232df-ed9d-403d-997b-881d762b462d TargetServiceName=null EntryTimestamp=2021-07-13T00:03:23.568Z InvocationID=f160719b-fbce-4e25-9e48-61da8baacedf ErrorCode=500 lock resource with id 279dd111-dc86-41d4-8ab6-a32289b1d5ac 2021-07-13T00:03:44.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a878c11-c958-4859-a922-8504d76ba175 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:03:44.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a878c11-c958-4859-a922-8504d76ba175 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6c7dc9b4 2021-07-13T00:03:44.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a878c11-c958-4859-a922-8504d76ba175 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:03:44.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a878c11-c958-4859-a922-8504d76ba175 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:03:45.875Z [qtp817348612-17] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VIP Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "220232df-ed9d-403d-997b-881d762b462d" INVARIANT_UUID = "c1ac5760-c02c-46e3-b34d-f1cac3149a9e" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "279dd111-dc86-41d4-8ab6-a32289b1d5ac.assettoscatemplate" CURR_ARTIFACT_UUID = "279dd111-dc86-41d4-8ab6-a32289b1d5ac.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VipCp-template.yml,0,279dd111-dc86-41d4-8ab6-a32289b1d5ac.assettoscatemplate,0 " STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=c8bb54ac-bf83-4105-8dea-65f7e398bac2 TargetServiceName=null EntryTimestamp=2021-07-13T00:03:23.568Z InvocationID=f160719b-fbce-4e25-9e48-61da8baacedf ErrorCode=500 Starting to create Csar for component VIP Cp 2021-07-13T00:03:46.211Z [qtp817348612-17] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VIP Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "220232df-ed9d-403d-997b-881d762b462d" INVARIANT_UUID = "c1ac5760-c02c-46e3-b34d-f1cac3149a9e" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "279dd111-dc86-41d4-8ab6-a32289b1d5ac.assettoscatemplate" CURR_ARTIFACT_UUID = "279dd111-dc86-41d4-8ab6-a32289b1d5ac.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VipCp-template.yml,0,279dd111-dc86-41d4-8ab6-a32289b1d5ac.assettoscatemplate,0 " STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=c8bb54ac-bf83-4105-8dea-65f7e398bac2 TargetServiceName=null EntryTimestamp=2021-07-13T00:03:23.568Z InvocationID=f160719b-fbce-4e25-9e48-61da8baacedf ErrorCode=500 Ended create Csar for component VIP Cp 2021-07-13T00:03:46.211Z [qtp817348612-17] INFO o.o.s.b.m.o.impl.GraphLockOperation AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VIP Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "220232df-ed9d-403d-997b-881d762b462d" INVARIANT_UUID = "c1ac5760-c02c-46e3-b34d-f1cac3149a9e" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "279dd111-dc86-41d4-8ab6-a32289b1d5ac.assettoscatemplate" CURR_ARTIFACT_UUID = "279dd111-dc86-41d4-8ab6-a32289b1d5ac.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VipCp-template.yml,0,279dd111-dc86-41d4-8ab6-a32289b1d5ac.assettoscatemplate,0 " STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=c8bb54ac-bf83-4105-8dea-65f7e398bac2 TargetServiceName=null EntryTimestamp=2021-07-13T00:03:23.568Z InvocationID=f160719b-fbce-4e25-9e48-61da8baacedf ErrorCode=500 lock resource with id 279dd111-dc86-41d4-8ab6-a32289b1d5ac 2021-07-13T00:03:48.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32e49d8d-3138-4515-8f2b-03b14854bda3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:03:48.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32e49d8d-3138-4515-8f2b-03b14854bda3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2f893363 2021-07-13T00:03:48.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32e49d8d-3138-4515-8f2b-03b14854bda3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:03:48.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32e49d8d-3138-4515-8f2b-03b14854bda3 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:03:49.377Z [qtp817348612-17] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VIP Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "220232df-ed9d-403d-997b-881d762b462d" INVARIANT_UUID = "c1ac5760-c02c-46e3-b34d-f1cac3149a9e" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "279dd111-dc86-41d4-8ab6-a32289b1d5ac.assettoscacsar" CURR_ARTIFACT_UUID = "279dd111-dc86-41d4-8ab6-a32289b1d5ac.assettoscacsar" ARTIFACT_DATA = "TOSCA,'assettoscacsar',TOSCA_CSAR,resource-VipCp-csar.csar,0,279dd111-dc86-41d4-8ab6-a32289b1d5ac.assettoscacsar,0 " STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR ResponseCode=201 RequestID=null ElapsedTime=25809 PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp StatusCode=COMPLETE TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=91faeca4-cca4-4c44-9263-71b948085d07 TargetServiceName=null EntryTimestamp=2021-07-13T00:03:23.568Z InvocationID=f160719b-fbce-4e25-9e48-61da8baacedf ErrorCode=500 LogTimestamp=2021-07-13T00:03:49.377124Z ResponseDesc=Created Exiting. 2021-07-13T00:03:49.377Z [qtp817348612-17] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 SC="201" 2021-07-13T00:03:49.466Z [qtp817348612-96] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=4a1792ea-30d5-435d-b2c0-d37dc73f6366 InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS RemoteHost=10.242.77.79 ServiceName=v1/catalog/upload/multipart ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:03:49.466041Z InvocationID=efa1e17b-ab68-48ae-930e-d85308ac6f43 LogTimestamp=2021-07-13T00:03:49.466081Z Entering 2021-07-13T00:03:49.466Z [qtp817348612-96] INFO o.o.sdc.be.filters.BeServletFilter RequestID=4a1792ea-30d5-435d-b2c0-d37dc73f6366 InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS userId=jh0003 RemoteHost=10.242.77.79 ServiceName=v1/catalog/upload/multipart ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:03:49.466041Z InvocationID=efa1e17b-ab68-48ae-930e-d85308ac6f43 LogTimestamp=2021-07-13T00:03:49.466081Z localAddr=10.242.98.212 remoteAddr=10.242.77.79 POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 2021-07-13T00:03:51.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0f49d4c-2598-45cf-a167-676b094332a3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:03:51.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0f49d4c-2598-45cf-a167-676b094332a3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4934c951 2021-07-13T00:03:51.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0f49d4c-2598-45cf-a167-676b094332a3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:03:51.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0f49d4c-2598-45cf-a167-676b094332a3 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:03:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:03:52.925Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:03:53.760Z [qtp817348612-96] INFO o.o.s.b.m.o.impl.GraphLockOperation RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:03:49.466Z InvocationID=57ccb1d5-3c5e-4a91-87b0-b87de5739ceb PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp lock resource with id 06bb209b-49eb-4226-8185-6fb868c96d23 2021-07-13T00:03:54.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8746f4ba-a615-4918-87d1-fcf525a26cc5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:03:54.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8746f4ba-a615-4918-87d1-fcf525a26cc5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1e7c8abd 2021-07-13T00:03:54.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8746f4ba-a615-4918-87d1-fcf525a26cc5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:03:54.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8746f4ba-a615-4918-87d1-fcf525a26cc5 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:03:55.866Z [qtp817348612-96] INFO o.o.s.b.c.l.LifecycleBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:03:49.466Z InvocationID=57ccb1d5-3c5e-4a91-87b0-b87de5739ceb PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp owner of resource 06bb209b-49eb-4226-8185-6fb868c96d23 is jh0003 2021-07-13T00:03:56.665Z [qtp817348612-96] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VDU VirtualBlockStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "47173899-e314-4643-8ab5-922b56430cc0" INVARIANT_UUID = "36636ba5-5424-44d1-bb58-6bbbc3158245" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:03:49.466Z InvocationID=57ccb1d5-3c5e-4a91-87b0-b87de5739ceb PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:03:56.666Z [qtp817348612-96] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VDU VirtualBlockStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "47173899-e314-4643-8ab5-922b56430cc0" INVARIANT_UUID = "36636ba5-5424-44d1-bb58-6bbbc3158245" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:03:49.466Z InvocationID=57ccb1d5-3c5e-4a91-87b0-b87de5739ceb ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:03:56.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d42e1ff2-22f1-499d-b0ed-c7e803a5243b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:03:56.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d42e1ff2-22f1-499d-b0ed-c7e803a5243b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c4ed0af 2021-07-13T00:03:56.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d42e1ff2-22f1-499d-b0ed-c7e803a5243b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:03:56.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d42e1ff2-22f1-499d-b0ed-c7e803a5243b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:04:00.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13531469-c3a3-45e1-ba16-7f945f1300bc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:04:00.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13531469-c3a3-45e1-ba16-7f945f1300bc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7abaea6e 2021-07-13T00:04:00.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13531469-c3a3-45e1-ba16-7f945f1300bc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:04:00.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13531469-c3a3-45e1-ba16-7f945f1300bc PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:04:03.168Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=282eff12-9762-418a-8846-924d803ca002 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:04:03.168Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=282eff12-9762-418a-8846-924d803ca002 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@152bca29 2021-07-13T00:04:03.168Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=282eff12-9762-418a-8846-924d803ca002 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:04:03.169Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=282eff12-9762-418a-8846-924d803ca002 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:04:03.660Z [qtp817348612-96] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VDU VirtualBlockStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "47173899-e314-4643-8ab5-922b56430cc0" INVARIANT_UUID = "36636ba5-5424-44d1-bb58-6bbbc3158245" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=b7124dbe-8895-48bd-97ff-9d7b054becc8 TargetServiceName=null EntryTimestamp=2021-07-13T00:03:49.466Z InvocationID=57ccb1d5-3c5e-4a91-87b0-b87de5739ceb ErrorCode=500 The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:04:03.661Z [qtp817348612-96] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VDU VirtualBlockStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "47173899-e314-4643-8ab5-922b56430cc0" INVARIANT_UUID = "36636ba5-5424-44d1-bb58-6bbbc3158245" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=b7124dbe-8895-48bd-97ff-9d7b054becc8 TargetServiceName=null EntryTimestamp=2021-07-13T00:03:49.466Z InvocationID=57ccb1d5-3c5e-4a91-87b0-b87de5739ceb ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:04:06.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad0c9dd6-237d-406c-90b0-1c4c7c0f3cc2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:04:06.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad0c9dd6-237d-406c-90b0-1c4c7c0f3cc2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6ff88353 2021-07-13T00:04:06.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad0c9dd6-237d-406c-90b0-1c4c7c0f3cc2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:04:06.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad0c9dd6-237d-406c-90b0-1c4c7c0f3cc2 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:04:07.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:04:07.460Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:04:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:04:07.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:04:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:04:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:04:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:04:09.911Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:04:10.073Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41d2cc57-485b-4a9c-a8a1-236c8bb50a4e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:04:10.073Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41d2cc57-485b-4a9c-a8a1-236c8bb50a4e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@559fdab 2021-07-13T00:04:10.073Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41d2cc57-485b-4a9c-a8a1-236c8bb50a4e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:04:10.073Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41d2cc57-485b-4a9c-a8a1-236c8bb50a4e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:04:10.575Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=41d2cc57-485b-4a9c-a8a1-236c8bb50a4e ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:04:10.575Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=41d2cc57-485b-4a9c-a8a1-236c8bb50a4e ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:04:10.673Z [qtp817348612-96] INFO o.o.s.b.m.o.impl.GraphLockOperation AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VDU VirtualBlockStorage" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "47173899-e314-4643-8ab5-922b56430cc0" INVARIANT_UUID = "36636ba5-5424-44d1-bb58-6bbbc3158245" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" STATUS = "200" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.Vdu.VirtualBlockStorage" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=b7124dbe-8895-48bd-97ff-9d7b054becc8 TargetServiceName=null EntryTimestamp=2021-07-13T00:03:49.466Z InvocationID=57ccb1d5-3c5e-4a91-87b0-b87de5739ceb ErrorCode=500 lock resource with id c9a68946-cd3f-41b6-b407-77202088514f 2021-07-13T00:04:11.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b52e8f47-6451-4a35-aa7e-71b7efa93f6a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:04:11.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b52e8f47-6451-4a35-aa7e-71b7efa93f6a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@289d9ef4 2021-07-13T00:04:11.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b52e8f47-6451-4a35-aa7e-71b7efa93f6a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:04:11.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b52e8f47-6451-4a35-aa7e-71b7efa93f6a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:04:11.960Z [qtp817348612-96] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VDU VirtualBlockStorage" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "47173899-e314-4643-8ab5-922b56430cc0" INVARIANT_UUID = "36636ba5-5424-44d1-bb58-6bbbc3158245" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" STATUS = "200" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.Vdu.VirtualBlockStorage" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=b7124dbe-8895-48bd-97ff-9d7b054becc8 TargetServiceName=null EntryTimestamp=2021-07-13T00:03:49.466Z InvocationID=57ccb1d5-3c5e-4a91-87b0-b87de5739ceb ErrorCode=500 owner of resource c9a68946-cd3f-41b6-b407-77202088514f is jh0003 2021-07-13T00:04:12.161Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b52e8f47-6451-4a35-aa7e-71b7efa93f6a ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:04:12.161Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b52e8f47-6451-4a35-aa7e-71b7efa93f6a ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:04:14.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74708d36-0654-4b15-96be-de06116dfe6a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:04:14.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74708d36-0654-4b15-96be-de06116dfe6a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63ac734c 2021-07-13T00:04:14.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74708d36-0654-4b15-96be-de06116dfe6a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:04:14.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74708d36-0654-4b15-96be-de06116dfe6a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:04:16.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:04:16.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:04:18.374Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=160a38e7-4d9d-447a-ac6c-b4ce9eebfb8f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:04:18.374Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=160a38e7-4d9d-447a-ac6c-b4ce9eebfb8f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@48659671 2021-07-13T00:04:18.374Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=160a38e7-4d9d-447a-ac6c-b4ce9eebfb8f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:04:18.374Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=160a38e7-4d9d-447a-ac6c-b4ce9eebfb8f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:04:18.760Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=160a38e7-4d9d-447a-ac6c-b4ce9eebfb8f ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:04:18.760Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=160a38e7-4d9d-447a-ac6c-b4ce9eebfb8f ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:04:18.763Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:04:21.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9cc22620-6bfb-4ded-95e3-c9b81115fad9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:04:21.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9cc22620-6bfb-4ded-95e3-c9b81115fad9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@29b9e73 2021-07-13T00:04:21.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9cc22620-6bfb-4ded-95e3-c9b81115fad9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:04:21.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9cc22620-6bfb-4ded-95e3-c9b81115fad9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:04:21.663Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9cc22620-6bfb-4ded-95e3-c9b81115fad9 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:04:21.663Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9cc22620-6bfb-4ded-95e3-c9b81115fad9 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:04:22.066Z [qtp817348612-96] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VDU VirtualBlockStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "b7124dbe-8895-48bd-97ff-9d7b054becc8" INVARIANT_UUID = "36636ba5-5424-44d1-bb58-6bbbc3158245" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=b7124dbe-8895-48bd-97ff-9d7b054becc8 TargetServiceName=null EntryTimestamp=2021-07-13T00:03:49.466Z InvocationID=57ccb1d5-3c5e-4a91-87b0-b87de5739ceb ErrorCode=500 owner of resource c9a68946-cd3f-41b6-b407-77202088514f is jh0003 2021-07-13T00:04:22.066Z [qtp817348612-96] INFO o.o.s.b.c.l.CertificationChangeTransition AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VDU VirtualBlockStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "b7124dbe-8895-48bd-97ff-9d7b054becc8" INVARIANT_UUID = "36636ba5-5424-44d1-bb58-6bbbc3158245" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=b7124dbe-8895-48bd-97ff-9d7b054becc8 TargetServiceName=null EntryTimestamp=2021-07-13T00:03:49.466Z InvocationID=57ccb1d5-3c5e-4a91-87b0-b87de5739ceb ErrorCode=500 validate before certification change. resource name=VDU VirtualBlockStorage, oldState=NOT_CERTIFIED_CHECKIN, owner userId=jh0003 2021-07-13T00:04:22.066Z [qtp817348612-96] INFO o.o.s.b.c.l.CertificationChangeTransition AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VDU VirtualBlockStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "b7124dbe-8895-48bd-97ff-9d7b054becc8" INVARIANT_UUID = "36636ba5-5424-44d1-bb58-6bbbc3158245" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=b7124dbe-8895-48bd-97ff-9d7b054becc8 TargetServiceName=null EntryTimestamp=2021-07-13T00:03:49.466Z InvocationID=57ccb1d5-3c5e-4a91-87b0-b87de5739ceb ErrorCode=500 start performing certification change for resource c9a68946-cd3f-41b6-b407-77202088514f 2021-07-13T00:04:22.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:04:22.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:04:23.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9cbadd9a-a7ff-47ec-9539-1ce843ec84db PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:04:23.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9cbadd9a-a7ff-47ec-9539-1ce843ec84db PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@46b5daea 2021-07-13T00:04:23.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9cbadd9a-a7ff-47ec-9539-1ce843ec84db PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:04:23.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9cbadd9a-a7ff-47ec-9539-1ce843ec84db PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:04:24.333Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:04:26.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15490a13-1644-477d-b236-3e468f1eb8e8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:04:26.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15490a13-1644-477d-b236-3e468f1eb8e8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4603e763 2021-07-13T00:04:26.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15490a13-1644-477d-b236-3e468f1eb8e8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:04:26.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15490a13-1644-477d-b236-3e468f1eb8e8 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:04:28.264Z [qtp817348612-96] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VDU VirtualBlockStorage" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "b7124dbe-8895-48bd-97ff-9d7b054becc8" INVARIANT_UUID = "36636ba5-5424-44d1-bb58-6bbbc3158245" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "06bb209b-49eb-4226-8185-6fb868c96d23.assettoscatemplate" CURR_ARTIFACT_UUID = "06bb209b-49eb-4226-8185-6fb868c96d23.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VduVirtualblockstorage-template.yml,0,06bb209b-49eb-4226-8185-6fb868c96d23.assettoscatemplate,1,da614974-2a9b-4717-bf37-bcb300de8c9e" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=20e26bc8-717a-4956-be4e-055f82cef499 TargetServiceName=null EntryTimestamp=2021-07-13T00:03:49.466Z InvocationID=57ccb1d5-3c5e-4a91-87b0-b87de5739ceb ErrorCode=500 Starting to create Csar for component VDU VirtualBlockStorage 2021-07-13T00:04:28.293Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01wxu4qpxyf9v1mkhluhlxdo74166 2021-07-13T00:04:28.293Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0nnl6zf7ir7lmbvefcxfsgwpn168 2021-07-13T00:04:28.293Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01leyq2fh4jrwd189srs63u3dkp174 2021-07-13T00:04:28.293Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0ubtcht5y8w5z16x7nru68qjp8186 2021-07-13T00:04:28.293Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0144qm8w4y2opahu1zy677hhfh189 2021-07-13T00:04:28.293Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01jen11uikd0ui1wsh1m47huxjn180 2021-07-13T00:04:28.293Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0loww5fm49n4k1l7b0653doohv182 2021-07-13T00:04:28.293Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01ci1jxret61ej39d3cgxqfw8e188 2021-07-13T00:04:28.293Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0hmg2mhy3r8zn1b12ziibsxa46184 2021-07-13T00:04:28.293Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node07t91z38kimwwxvafkkgawdyq151 2021-07-13T00:04:28.294Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0lmxugzoodeva17v0n51qgthqw190 2021-07-13T00:04:28.294Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node011ry7o65ophn8z3igl63rwv5q185 2021-07-13T00:04:28.294Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01cm85x2aynt6ozl08pivxc1b183 2021-07-13T00:04:28.294Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01simiagij01nqoc8wag20sddd165 2021-07-13T00:04:28.294Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node019lv0jdq82zhb14nxzj447bxh4187 2021-07-13T00:04:28.294Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0fv1pzk8n8nlv1kh4rxuoo4hmi179 2021-07-13T00:04:28.294Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node039ke1aav3ox914mabd6cjnfm8157 2021-07-13T00:04:28.294Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0l65z37k6iyxr188pm8gt2vtl3159 2021-07-13T00:04:28.294Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0xpdxkece3rwqmhwq4cbm3js9176 2021-07-13T00:04:28.294Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01emnyswotfrwym54mqka3oig1175 2021-07-13T00:04:28.294Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0dvxpsedg691p1fgos3m3vsngy178 2021-07-13T00:04:28.294Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0101lr37m2bmyp1rklsvfm27oxc164 2021-07-13T00:04:28.294Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0pz1icu8qpr3t127oljy1u1jq8171 2021-07-13T00:04:28.294Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0fldnrwdhtyb6vmwza3yk4wr8167 2021-07-13T00:04:28.294Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0q1jal7vow25mx7dgt8oocfdb154 2021-07-13T00:04:28.295Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0dq2blnpgo5ls12coenf86lcvb153 2021-07-13T00:04:28.295Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01muw77a8liczu1xqwnef3piahd161 2021-07-13T00:04:28.295Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01maj9ajux2mp1faahl0p54rld172 2021-07-13T00:04:28.295Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0nsjohpi4hygs1jgnoxy594jbc169 2021-07-13T00:04:28.295Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01ran9hf09dbfyueozgphpqxvv160 2021-07-13T00:04:28.295Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node017sndrpai7djnr5tmink9dx3a177 2021-07-13T00:04:28.295Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0j1x5r879vofbmb5u4g309w51163 2021-07-13T00:04:28.295Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0yfjv76zvaxkz1e00wk37ovnie156 2021-07-13T00:04:28.295Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0eo5oj20z12uhjwionkfoy4qh155 2021-07-13T00:04:28.295Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0vmato3lbb5ue1u9js6hiq0kn158 2021-07-13T00:04:28.295Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01vbkst83a3zr115y16lzwjlnv4162 2021-07-13T00:04:28.295Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0yk7m0y05ecdi13vazgs0vgzf2181 2021-07-13T00:04:28.295Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0ri66u11tlad71s9pyk4gg8qhu170 2021-07-13T00:04:28.295Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01wec98dyrncaixdznzmam1t1173 2021-07-13T00:04:28.573Z [qtp817348612-96] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VDU VirtualBlockStorage" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "b7124dbe-8895-48bd-97ff-9d7b054becc8" INVARIANT_UUID = "36636ba5-5424-44d1-bb58-6bbbc3158245" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "06bb209b-49eb-4226-8185-6fb868c96d23.assettoscatemplate" CURR_ARTIFACT_UUID = "06bb209b-49eb-4226-8185-6fb868c96d23.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VduVirtualblockstorage-template.yml,0,06bb209b-49eb-4226-8185-6fb868c96d23.assettoscatemplate,1,da614974-2a9b-4717-bf37-bcb300de8c9e" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=20e26bc8-717a-4956-be4e-055f82cef499 TargetServiceName=null EntryTimestamp=2021-07-13T00:03:49.466Z InvocationID=57ccb1d5-3c5e-4a91-87b0-b87de5739ceb ErrorCode=500 Ended create Csar for component VDU VirtualBlockStorage 2021-07-13T00:04:29.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0decaf15-8eb8-4e38-8507-76d45c6faa14 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:04:29.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0decaf15-8eb8-4e38-8507-76d45c6faa14 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@223127cc 2021-07-13T00:04:29.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0decaf15-8eb8-4e38-8507-76d45c6faa14 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:04:29.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0decaf15-8eb8-4e38-8507-76d45c6faa14 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:04:32.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d63d0a69-6aab-41a9-923a-d67589967387 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:04:32.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d63d0a69-6aab-41a9-923a-d67589967387 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7c075dff 2021-07-13T00:04:32.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d63d0a69-6aab-41a9-923a-d67589967387 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:04:32.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d63d0a69-6aab-41a9-923a-d67589967387 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:04:35.859Z [qtp817348612-96] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU VirtualBlockStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "b7124dbe-8895-48bd-97ff-9d7b054becc8" INVARIANT_UUID = "36636ba5-5424-44d1-bb58-6bbbc3158245" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=4311c349-b0be-45d2-b137-8e3fa1ff37e6 TargetServiceName=null EntryTimestamp=2021-07-13T00:03:49.466Z InvocationID=57ccb1d5-3c5e-4a91-87b0-b87de5739ceb ErrorCode=500 The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:04:35.860Z [qtp817348612-96] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU VirtualBlockStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "b7124dbe-8895-48bd-97ff-9d7b054becc8" INVARIANT_UUID = "36636ba5-5424-44d1-bb58-6bbbc3158245" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=4311c349-b0be-45d2-b137-8e3fa1ff37e6 TargetServiceName=null EntryTimestamp=2021-07-13T00:03:49.466Z InvocationID=57ccb1d5-3c5e-4a91-87b0-b87de5739ceb ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:04:35.860Z [qtp817348612-96] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU VirtualBlockStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "b7124dbe-8895-48bd-97ff-9d7b054becc8" INVARIANT_UUID = "36636ba5-5424-44d1-bb58-6bbbc3158245" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=4311c349-b0be-45d2-b137-8e3fa1ff37e6 TargetServiceName=null EntryTimestamp=2021-07-13T00:03:49.466Z InvocationID=57ccb1d5-3c5e-4a91-87b0-b87de5739ceb ErrorCode=500 unlock component c9a68946-cd3f-41b6-b407-77202088514f 2021-07-13T00:04:35.860Z [qtp817348612-96] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU VirtualBlockStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "b7124dbe-8895-48bd-97ff-9d7b054becc8" INVARIANT_UUID = "36636ba5-5424-44d1-bb58-6bbbc3158245" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=4311c349-b0be-45d2-b137-8e3fa1ff37e6 TargetServiceName=null EntryTimestamp=2021-07-13T00:03:49.466Z InvocationID=57ccb1d5-3c5e-4a91-87b0-b87de5739ceb ErrorCode=500 During change state, another component c9a68946-cd3f-41b6-b407-77202088514f has been created/updated 2021-07-13T00:04:35.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03d6b17a-a581-4417-8fd8-10b7f1cdc107 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:04:35.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03d6b17a-a581-4417-8fd8-10b7f1cdc107 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@140daa1f 2021-07-13T00:04:35.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03d6b17a-a581-4417-8fd8-10b7f1cdc107 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:04:35.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03d6b17a-a581-4417-8fd8-10b7f1cdc107 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:04:37.571Z [qtp817348612-96] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU VirtualBlockStorage" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "b7124dbe-8895-48bd-97ff-9d7b054becc8" INVARIANT_UUID = "36636ba5-5424-44d1-bb58-6bbbc3158245" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR ResponseCode=201 RequestID=null ElapsedTime=48104 PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp StatusCode=COMPLETE TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=4311c349-b0be-45d2-b137-8e3fa1ff37e6 TargetServiceName=null EntryTimestamp=2021-07-13T00:03:49.466Z InvocationID=57ccb1d5-3c5e-4a91-87b0-b87de5739ceb ErrorCode=500 LogTimestamp=2021-07-13T00:04:37.570409Z ResponseDesc=Created Exiting. 2021-07-13T00:04:37.571Z [qtp817348612-96] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 SC="201" 2021-07-13T00:04:37.816Z [qtp817348612-101] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=b7591e38-624d-46ed-a016-13609371e118 InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS RemoteHost=10.242.77.79 ServiceName=v1/catalog/upload/multipart ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:04:37.815886Z InvocationID=09bd4315-e819-47d2-9661-204dd8ca28a3 LogTimestamp=2021-07-13T00:04:37.815952Z Entering 2021-07-13T00:04:37.816Z [qtp817348612-101] INFO o.o.sdc.be.filters.BeServletFilter RequestID=b7591e38-624d-46ed-a016-13609371e118 InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS userId=jh0003 RemoteHost=10.242.77.79 ServiceName=v1/catalog/upload/multipart ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:04:37.815886Z InvocationID=09bd4315-e819-47d2-9661-204dd8ca28a3 LogTimestamp=2021-07-13T00:04:37.815952Z localAddr=10.242.98.212 remoteAddr=10.242.77.79 POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 2021-07-13T00:04:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:04:37.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:04:38.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=879fd44f-08c5-4a9c-a51a-423fa8b1c9c2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:04:38.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=879fd44f-08c5-4a9c-a51a-423fa8b1c9c2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4d6c7f02 2021-07-13T00:04:38.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=879fd44f-08c5-4a9c-a51a-423fa8b1c9c2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:04:38.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=879fd44f-08c5-4a9c-a51a-423fa8b1c9c2 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:04:41.962Z [qtp817348612-101] INFO o.o.s.b.m.o.impl.GraphLockOperation RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:04:37.816Z InvocationID=fbad2982-063b-4b76-aeb0-5fa5ee551b78 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp lock resource with id 2a1c6b8e-024a-4f8c-8456-8cef2de565e0 2021-07-13T00:04:42.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=16434f7d-0263-42ae-aec3-8d476dcfd295 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:04:42.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=16434f7d-0263-42ae-aec3-8d476dcfd295 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@66482e24 2021-07-13T00:04:42.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=16434f7d-0263-42ae-aec3-8d476dcfd295 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:04:42.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=16434f7d-0263-42ae-aec3-8d476dcfd295 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:04:44.211Z [qtp817348612-101] INFO o.o.s.b.c.l.LifecycleBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:04:37.816Z InvocationID=fbad2982-063b-4b76-aeb0-5fa5ee551b78 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp owner of resource 2a1c6b8e-024a-4f8c-8456-8cef2de565e0 is jh0003 2021-07-13T00:04:44.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78abf875-d513-4d11-b5dd-026a2588beb6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:04:44.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78abf875-d513-4d11-b5dd-026a2588beb6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@20667fb7 2021-07-13T00:04:44.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78abf875-d513-4d11-b5dd-026a2588beb6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:04:44.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78abf875-d513-4d11-b5dd-026a2588beb6 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:04:45.159Z [qtp817348612-101] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "c6d2be33-73bb-4fad-a4cf-ba57c5f20ef9" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:04:37.816Z InvocationID=fbad2982-063b-4b76-aeb0-5fa5ee551b78 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:04:45.159Z [qtp817348612-101] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "c6d2be33-73bb-4fad-a4cf-ba57c5f20ef9" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:04:37.816Z InvocationID=fbad2982-063b-4b76-aeb0-5fa5ee551b78 ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:04:48.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f28d4bcf-f0e9-4d47-8525-ef4967d672a2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:04:48.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f28d4bcf-f0e9-4d47-8525-ef4967d672a2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@59c274f6 2021-07-13T00:04:48.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f28d4bcf-f0e9-4d47-8525-ef4967d672a2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:04:48.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f28d4bcf-f0e9-4d47-8525-ef4967d672a2 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:04:50.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ef86848a-9890-4d1a-9cdc-ec997a371f21 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:04:50.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ef86848a-9890-4d1a-9cdc-ec997a371f21 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@77f894a3 2021-07-13T00:04:50.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ef86848a-9890-4d1a-9cdc-ec997a371f21 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:04:50.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ef86848a-9890-4d1a-9cdc-ec997a371f21 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:04:51.461Z [qtp817348612-101] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "c6d2be33-73bb-4fad-a4cf-ba57c5f20ef9" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=10b1b941-7f72-4723-ac16-250117ebd8e3 TargetServiceName=null EntryTimestamp=2021-07-13T00:04:37.816Z InvocationID=fbad2982-063b-4b76-aeb0-5fa5ee551b78 ErrorCode=500 The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:04:51.461Z [qtp817348612-101] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "c6d2be33-73bb-4fad-a4cf-ba57c5f20ef9" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=10b1b941-7f72-4723-ac16-250117ebd8e3 TargetServiceName=null EntryTimestamp=2021-07-13T00:04:37.816Z InvocationID=fbad2982-063b-4b76-aeb0-5fa5ee551b78 ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:04:52.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:04:52.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:04:54.267Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b49a6adb-ead5-46d4-bf6b-7be9a4d79fbb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:04:54.267Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b49a6adb-ead5-46d4-bf6b-7be9a4d79fbb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5066ad76 2021-07-13T00:04:54.267Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b49a6adb-ead5-46d4-bf6b-7be9a4d79fbb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:04:54.267Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b49a6adb-ead5-46d4-bf6b-7be9a4d79fbb PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:04:56.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de762cd4-952b-4eef-b22c-d367bc40728f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:04:56.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de762cd4-952b-4eef-b22c-d367bc40728f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@43eeeb0a 2021-07-13T00:04:56.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de762cd4-952b-4eef-b22c-d367bc40728f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:04:56.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de762cd4-952b-4eef-b22c-d367bc40728f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:04:58.771Z [qtp817348612-101] INFO o.o.s.b.m.o.impl.GraphLockOperation AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "c6d2be33-73bb-4fad-a4cf-ba57c5f20ef9" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" STATUS = "200" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.VNF" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=10b1b941-7f72-4723-ac16-250117ebd8e3 TargetServiceName=null EntryTimestamp=2021-07-13T00:04:37.816Z InvocationID=fbad2982-063b-4b76-aeb0-5fa5ee551b78 ErrorCode=500 lock resource with id 6a1a5cee-acdb-4880-aa85-7a5c74cbad9a 2021-07-13T00:04:59.571Z [qtp817348612-101] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "c6d2be33-73bb-4fad-a4cf-ba57c5f20ef9" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" STATUS = "200" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.VNF" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=10b1b941-7f72-4723-ac16-250117ebd8e3 TargetServiceName=null EntryTimestamp=2021-07-13T00:04:37.816Z InvocationID=fbad2982-063b-4b76-aeb0-5fa5ee551b78 ErrorCode=500 owner of resource 6a1a5cee-acdb-4880-aa85-7a5c74cbad9a is jh0003 2021-07-13T00:04:59.874Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b19bb72f-528b-4125-a593-03c1b06676df PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:04:59.874Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b19bb72f-528b-4125-a593-03c1b06676df PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@9415245 2021-07-13T00:04:59.875Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b19bb72f-528b-4125-a593-03c1b06676df PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:04:59.875Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b19bb72f-528b-4125-a593-03c1b06676df PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:05:02.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e037e008-2887-414c-9f0a-de0edc3e1d72 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:05:02.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e037e008-2887-414c-9f0a-de0edc3e1d72 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3aaf8242 2021-07-13T00:05:02.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e037e008-2887-414c-9f0a-de0edc3e1d72 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:05:02.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e037e008-2887-414c-9f0a-de0edc3e1d72 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:05:05.872Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5cc9858-06cb-4f39-beeb-49e3ad884a33 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:05:05.872Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5cc9858-06cb-4f39-beeb-49e3ad884a33 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f47857c 2021-07-13T00:05:05.872Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5cc9858-06cb-4f39-beeb-49e3ad884a33 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:05:05.872Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5cc9858-06cb-4f39-beeb-49e3ad884a33 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:05:07.311Z [qtp817348612-101] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=10b1b941-7f72-4723-ac16-250117ebd8e3 TargetServiceName=null EntryTimestamp=2021-07-13T00:04:37.816Z InvocationID=fbad2982-063b-4b76-aeb0-5fa5ee551b78 ErrorCode=500 owner of resource 6a1a5cee-acdb-4880-aa85-7a5c74cbad9a is jh0003 2021-07-13T00:05:07.312Z [qtp817348612-101] INFO o.o.s.b.c.l.CertificationChangeTransition AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=10b1b941-7f72-4723-ac16-250117ebd8e3 TargetServiceName=null EntryTimestamp=2021-07-13T00:04:37.816Z InvocationID=fbad2982-063b-4b76-aeb0-5fa5ee551b78 ErrorCode=500 validate before certification change. resource name=VNF, oldState=NOT_CERTIFIED_CHECKIN, owner userId=jh0003 2021-07-13T00:05:07.312Z [qtp817348612-101] INFO o.o.s.b.c.l.CertificationChangeTransition AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=10b1b941-7f72-4723-ac16-250117ebd8e3 TargetServiceName=null EntryTimestamp=2021-07-13T00:04:37.816Z InvocationID=fbad2982-063b-4b76-aeb0-5fa5ee551b78 ErrorCode=500 start performing certification change for resource 6a1a5cee-acdb-4880-aa85-7a5c74cbad9a 2021-07-13T00:05:07.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:05:07.932Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:05:09.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=019a1a97-ff7b-4447-9b47-f57d19fbb2d9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:05:09.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=019a1a97-ff7b-4447-9b47-f57d19fbb2d9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6b458325 2021-07-13T00:05:09.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=019a1a97-ff7b-4447-9b47-f57d19fbb2d9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:05:09.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=019a1a97-ff7b-4447-9b47-f57d19fbb2d9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:05:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:05:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:05:09.369Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:05:11.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c2e9d7a1-8f3f-470d-8d2f-80a5a1c114c4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:05:11.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c2e9d7a1-8f3f-470d-8d2f-80a5a1c114c4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@68953772 2021-07-13T00:05:11.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c2e9d7a1-8f3f-470d-8d2f-80a5a1c114c4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:05:11.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c2e9d7a1-8f3f-470d-8d2f-80a5a1c114c4 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:05:12.857Z [qtp817348612-101] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate" CURR_ARTIFACT_UUID = "2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-Vnf-template.yml,0,2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate,1,1e0a017e-4724-4a07-aebd-f25962197db9" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=ceef1c8e-43fe-4f98-9afe-f7a8c496de28 TargetServiceName=null EntryTimestamp=2021-07-13T00:04:37.816Z InvocationID=fbad2982-063b-4b76-aeb0-5fa5ee551b78 ErrorCode=500 Starting to create Csar for component VNF 2021-07-13T00:05:12.911Z [qtp817348612-101] WARN o.o.s.e.AbstractSdncException AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate" CURR_ARTIFACT_UUID = "2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-Vnf-template.yml,0,2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate,1,1e0a017e-4724-4a07-aebd-f25962197db9" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity= ErrorCategory=WARN ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=ceef1c8e-43fe-4f98-9afe-f7a8c496de28 TargetServiceName=null EntryTimestamp=2021-07-13T00:04:37.816Z InvocationID=fbad2982-063b-4b76-aeb0-5fa5ee551b78 ErrorCode=900 Received less parameters than expected for error with messageId SVC4063, expected: 1, actual: 0. Missing parameters are padded with null values. 2021-07-13T00:05:12.913Z [qtp817348612-101] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate" CURR_ARTIFACT_UUID = "2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-Vnf-template.yml,0,2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate,1,1e0a017e-4724-4a07-aebd-f25962197db9" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Change LifecycleState ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=ceef1c8e-43fe-4f98-9afe-f7a8c496de28 TargetServiceName=null EntryTimestamp=2021-07-13T00:04:37.816Z InvocationID=fbad2982-063b-4b76-aeb0-5fa5ee551b78 ErrorCode=500 Operation towards database failed. 2021-07-13T00:05:12.913Z [qtp817348612-101] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate" CURR_ARTIFACT_UUID = "2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-Vnf-template.yml,0,2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate,1,1e0a017e-4724-4a07-aebd-f25962197db9" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Change LifecycleState ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=ceef1c8e-43fe-4f98-9afe-f7a8c496de28 TargetServiceName=null EntryTimestamp=2021-07-13T00:04:37.816Z InvocationID=fbad2982-063b-4b76-aeb0-5fa5ee551b78 ErrorCode=500 unlock component 6a1a5cee-acdb-4880-aa85-7a5c74cbad9a 2021-07-13T00:05:12.913Z [qtp817348612-101] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate" CURR_ARTIFACT_UUID = "2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-Vnf-template.yml,0,2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate,1,1e0a017e-4724-4a07-aebd-f25962197db9" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Change LifecycleState ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=ceef1c8e-43fe-4f98-9afe-f7a8c496de28 TargetServiceName=null EntryTimestamp=2021-07-13T00:04:37.816Z InvocationID=fbad2982-063b-4b76-aeb0-5fa5ee551b78 ErrorCode=500 During change state, another component 6a1a5cee-acdb-4880-aa85-7a5c74cbad9a has been created/updated 2021-07-13T00:05:14.411Z [qtp817348612-101] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate" CURR_ARTIFACT_UUID = "2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-Vnf-template.yml,0,2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate,1,1e0a017e-4724-4a07-aebd-f25962197db9" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Change LifecycleState - Certify ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=ceef1c8e-43fe-4f98-9afe-f7a8c496de28 TargetServiceName=null EntryTimestamp=2021-07-13T00:04:37.816Z InvocationID=fbad2982-063b-4b76-aeb0-5fa5ee551b78 ErrorCode=500 Unexpected error during operation 2021-07-13T00:05:14.412Z [qtp817348612-101] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate" CURR_ARTIFACT_UUID = "2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-Vnf-template.yml,0,2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate,1,1e0a017e-4724-4a07-aebd-f25962197db9" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Import Resource VNF.yml ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=ceef1c8e-43fe-4f98-9afe-f7a8c496de28 TargetServiceName=null EntryTimestamp=2021-07-13T00:04:37.816Z InvocationID=fbad2982-063b-4b76-aeb0-5fa5ee551b78 ErrorCode=500 Unexpected error during operation 2021-07-13T00:05:14.460Z [qtp817348612-101] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "" INVARIANT_UUID = "" PREV_VERSION = "1.0" CURR_VERSION = "" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "" STATUS = "404" DESC = "SVC4063: Error: Requested 'null' resource was not found." TOSCA_NODE_TYPE = "" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Upload Resource ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=ceef1c8e-43fe-4f98-9afe-f7a8c496de28 TargetServiceName=null EntryTimestamp=2021-07-13T00:04:37.816Z InvocationID=fbad2982-063b-4b76-aeb0-5fa5ee551b78 ErrorCode=900 Unexpected error during BE REST API execution 2021-07-13T00:05:14.464Z [qtp817348612-101] ERROR o.o.s.b.s.ResourceUploadServlet AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "" INVARIANT_UUID = "" PREV_VERSION = "1.0" CURR_VERSION = "" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "" STATUS = "404" DESC = "SVC4063: Error: Requested 'null' resource was not found." TOSCA_NODE_TYPE = "" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Upload Resource ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=ceef1c8e-43fe-4f98-9afe-f7a8c496de28 TargetServiceName=null EntryTimestamp=2021-07-13T00:04:37.816Z InvocationID=fbad2982-063b-4b76-aeb0-5fa5ee551b78 ErrorCode=900 #uploadMultipart('multipart', /tmp/rep10521323357250130994tmp, 'form-data; filename="VNF.zip"; name="resourceZip"', '{"payloadName": "VNF.yml", "contactId": "jh0003", "name": "VNF", "description": "VNF", "resourceIconPath": "compute", "resourceType": "VFC", "categories": [{"name": "Generic", "subcategories": [{"name": "Infrastructure"}]}], "vendorName": "ETSI", "vendorRelease": "3.3.1", "tags": ["VNF"]}', 'Request(POST https://sdc-be.onap:8443/sdc2/rest/v1/catalog/upload/multipart?createNewVersion=true)@62228854', 'jh0003', true): thrown org.openecomp.sdc.be.components.impl.exceptions.ByResponseFormatComponentException out of org.openecomp.sdc.be.components.impl.ArtifactsBusinessLogic#lambda$decodeToscaArtifactPayload$5[371] in 37s 2021-07-13T00:05:14.467Z [qtp817348612-101] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "" INVARIANT_UUID = "" PREV_VERSION = "1.0" CURR_VERSION = "" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "" STATUS = "404" DESC = "SVC4063: Error: Requested 'null' resource was not found." TOSCA_NODE_TYPE = "" SupportablityAction=GENERATE CSAR ResponseCode=404 RequestID=null ElapsedTime=36650 PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp StatusCode=ERROR TargetEntity=Upload Resource ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=ceef1c8e-43fe-4f98-9afe-f7a8c496de28 TargetServiceName=null ErrorDesc=Not Found EntryTimestamp=2021-07-13T00:04:37.816Z InvocationID=fbad2982-063b-4b76-aeb0-5fa5ee551b78 ErrorCode=404 LogTimestamp=2021-07-13T00:05:14.466925Z ResponseDesc=Not Found Exiting. 2021-07-13T00:05:14.468Z [qtp817348612-101] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 SC="404" 2021-07-13T00:05:14.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a90c4ac-d7e3-42df-82bf-ba2db2562dee PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:05:14.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a90c4ac-d7e3-42df-82bf-ba2db2562dee PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2eee05b0 2021-07-13T00:05:14.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a90c4ac-d7e3-42df-82bf-ba2db2562dee PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:05:14.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a90c4ac-d7e3-42df-82bf-ba2db2562dee PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:05:17.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=590ec378-7a91-42ab-92c1-f6db7fd3c5b7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:05:17.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=590ec378-7a91-42ab-92c1-f6db7fd3c5b7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4de10974 2021-07-13T00:05:17.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=590ec378-7a91-42ab-92c1-f6db7fd3c5b7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:05:17.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=590ec378-7a91-42ab-92c1-f6db7fd3c5b7 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:05:21.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1496c7fd-3ce5-4c11-8da5-8776fc5121a5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:05:21.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1496c7fd-3ce5-4c11-8da5-8776fc5121a5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@a7af746 2021-07-13T00:05:21.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1496c7fd-3ce5-4c11-8da5-8776fc5121a5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:05:21.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1496c7fd-3ce5-4c11-8da5-8776fc5121a5 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:05:22.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:05:22.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:05:23.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4997c26b-f062-449f-a425-563e945029b4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:05:23.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4997c26b-f062-449f-a425-563e945029b4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4b156fdc 2021-07-13T00:05:23.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4997c26b-f062-449f-a425-563e945029b4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:05:23.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4997c26b-f062-449f-a425-563e945029b4 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:05:24.410Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:05:24.566Z [qtp817348612-96] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=56636edc-84f1-4586-b8a5-88dac3187130 InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS RemoteHost=10.242.77.79 ServiceName=v1/catalog/upload/multipart ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:05:24.566147Z InvocationID=b6c99793-6914-46ef-80d5-0880545b099a LogTimestamp=2021-07-13T00:05:24.566257Z Entering 2021-07-13T00:05:24.566Z [qtp817348612-96] INFO o.o.sdc.be.filters.BeServletFilter RequestID=56636edc-84f1-4586-b8a5-88dac3187130 InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS userId=jh0003 RemoteHost=10.242.77.79 ServiceName=v1/catalog/upload/multipart ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:05:24.566147Z InvocationID=b6c99793-6914-46ef-80d5-0880545b099a LogTimestamp=2021-07-13T00:05:24.566257Z localAddr=10.242.98.212 remoteAddr=10.242.77.79 POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 2021-07-13T00:05:26.776Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53cc30fb-1505-4d29-ac91-53251e78760d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:05:26.776Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53cc30fb-1505-4d29-ac91-53251e78760d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1588308b 2021-07-13T00:05:26.776Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53cc30fb-1505-4d29-ac91-53251e78760d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:05:26.776Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53cc30fb-1505-4d29-ac91-53251e78760d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:05:29.815Z [qtp817348612-96] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Import Resource VNF.yml ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:05:24.566Z InvocationID=39f38ffa-3aa2-4735-901c-b9525f8bc5e9 ErrorCode=500 Unexpected error during operation 2021-07-13T00:05:29.820Z [qtp817348612-96] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "" INVARIANT_UUID = "" PREV_VERSION = "1.0" CURR_VERSION = "" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "" STATUS = "409" DESC = "SVC4083: Error: Version of this VNF was already promoted." TOSCA_NODE_TYPE = "" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Upload Resource ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:05:24.566Z InvocationID=39f38ffa-3aa2-4735-901c-b9525f8bc5e9 ErrorCode=900 Unexpected error during BE REST API execution 2021-07-13T00:05:29.821Z [qtp817348612-96] ERROR o.o.s.b.s.ResourceUploadServlet AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "" INVARIANT_UUID = "" PREV_VERSION = "1.0" CURR_VERSION = "" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "" STATUS = "409" DESC = "SVC4083: Error: Version of this VNF was already promoted." TOSCA_NODE_TYPE = "" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Upload Resource ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:05:24.566Z InvocationID=39f38ffa-3aa2-4735-901c-b9525f8bc5e9 ErrorCode=900 #uploadMultipart('multipart', /tmp/rep3024510717365052545tmp, 'form-data; filename="VNF.zip"; name="resourceZip"', '{"payloadName": "VNF.yml", "contactId": "jh0003", "name": "VNF", "description": "VNF", "resourceIconPath": "compute", "resourceType": "VFC", "categories": [{"name": "Generic", "subcategories": [{"name": "Infrastructure"}]}], "vendorName": "ETSI", "vendorRelease": "3.3.1", "tags": ["VNF"]}', 'Request(POST https://sdc-be.onap:8443/sdc2/rest/v1/catalog/upload/multipart?createNewVersion=true)@62228854', 'jh0003', true): thrown org.openecomp.sdc.be.components.impl.exceptions.ByActionStatusComponentException out of org.openecomp.sdc.be.components.impl.ResourceImportManager#importCertifiedResource[184] in 5s 2021-07-13T00:05:29.823Z [qtp817348612-96] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "" INVARIANT_UUID = "" PREV_VERSION = "1.0" CURR_VERSION = "" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "" STATUS = "409" DESC = "SVC4083: Error: Version of this VNF was already promoted." TOSCA_NODE_TYPE = "" ResponseCode=409 RequestID=null ElapsedTime=5256 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp StatusCode=ERROR TargetEntity=Upload Resource ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null ErrorDesc=Conflict EntryTimestamp=2021-07-13T00:05:24.566Z InvocationID=39f38ffa-3aa2-4735-901c-b9525f8bc5e9 ErrorCode=409 LogTimestamp=2021-07-13T00:05:29.822839Z ResponseDesc=Conflict Exiting. 2021-07-13T00:05:29.823Z [qtp817348612-96] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 SC="409" 2021-07-13T00:05:30.016Z [qtp817348612-101] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=d9ba2393-1a1f-4cb5-9314-4ee1c46d9cf6 InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS RemoteHost=10.242.77.79 ServiceName=v1/catalog/upload/multipart ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:05:30.016711Z InvocationID=52ecc600-1852-4d55-94e4-ea2e93d264b9 LogTimestamp=2021-07-13T00:05:30.016785Z Entering 2021-07-13T00:05:30.017Z [qtp817348612-101] INFO o.o.sdc.be.filters.BeServletFilter RequestID=d9ba2393-1a1f-4cb5-9314-4ee1c46d9cf6 InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS userId=jh0003 RemoteHost=10.242.77.79 ServiceName=v1/catalog/upload/multipart ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:05:30.016711Z InvocationID=52ecc600-1852-4d55-94e4-ea2e93d264b9 LogTimestamp=2021-07-13T00:05:30.016785Z localAddr=10.242.98.212 remoteAddr=10.242.77.79 POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 2021-07-13T00:05:30.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=57df47c8-ea8e-4c86-88db-02359ce31764 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:05:30.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=57df47c8-ea8e-4c86-88db-02359ce31764 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@506721e7 2021-07-13T00:05:30.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=57df47c8-ea8e-4c86-88db-02359ce31764 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:05:30.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=57df47c8-ea8e-4c86-88db-02359ce31764 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:05:33.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06e30344-c187-4a74-8a10-3cc0f87f3d9d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:05:33.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06e30344-c187-4a74-8a10-3cc0f87f3d9d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@dfa3c6a 2021-07-13T00:05:33.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06e30344-c187-4a74-8a10-3cc0f87f3d9d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:05:33.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06e30344-c187-4a74-8a10-3cc0f87f3d9d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:05:34.511Z [qtp817348612-101] INFO o.o.s.b.m.o.impl.GraphLockOperation RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:05:30.017Z InvocationID=953eb0db-7c0d-47a9-b241-314873727cbe PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp lock resource with id a52a7305-3368-4f9b-85fb-e7adaf38df1e 2021-07-13T00:05:35.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed350f4c-f3c3-47c9-8f87-bdd24c29ff61 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:05:35.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed350f4c-f3c3-47c9-8f87-bdd24c29ff61 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6df1c986 2021-07-13T00:05:35.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed350f4c-f3c3-47c9-8f87-bdd24c29ff61 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:05:35.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed350f4c-f3c3-47c9-8f87-bdd24c29ff61 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:05:36.812Z [qtp817348612-101] INFO o.o.s.b.c.l.LifecycleBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:05:30.017Z InvocationID=953eb0db-7c0d-47a9-b241-314873727cbe PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp owner of resource a52a7305-3368-4f9b-85fb-e7adaf38df1e is jh0003 2021-07-13T00:05:37.667Z [qtp817348612-101] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "NS" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "e7a6d00a-9316-47af-81a2-1583055b2f35" INVARIANT_UUID = "78ff1a10-5eed-4eab-9611-ac77e6cb368e" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:05:30.017Z InvocationID=953eb0db-7c0d-47a9-b241-314873727cbe PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:05:37.669Z [qtp817348612-101] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "NS" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "e7a6d00a-9316-47af-81a2-1583055b2f35" INVARIANT_UUID = "78ff1a10-5eed-4eab-9611-ac77e6cb368e" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:05:30.017Z InvocationID=953eb0db-7c0d-47a9-b241-314873727cbe ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:05:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:05:37.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:05:38.972Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28c9fc78-de12-41cb-9339-9950141e589f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:05:38.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28c9fc78-de12-41cb-9339-9950141e589f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3929b0db 2021-07-13T00:05:38.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28c9fc78-de12-41cb-9339-9950141e589f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:05:38.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28c9fc78-de12-41cb-9339-9950141e589f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:05:42.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5eb8babb-d085-48ed-9c83-8b7bd04e8cf4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:05:42.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5eb8babb-d085-48ed-9c83-8b7bd04e8cf4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@50a8c600 2021-07-13T00:05:42.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5eb8babb-d085-48ed-9c83-8b7bd04e8cf4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:05:42.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5eb8babb-d085-48ed-9c83-8b7bd04e8cf4 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:05:43.560Z [qtp817348612-101] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "NS" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "e7a6d00a-9316-47af-81a2-1583055b2f35" INVARIANT_UUID = "78ff1a10-5eed-4eab-9611-ac77e6cb368e" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=919f9a12-dde7-4248-8957-a6b22734ef89 TargetServiceName=null EntryTimestamp=2021-07-13T00:05:30.017Z InvocationID=953eb0db-7c0d-47a9-b241-314873727cbe ErrorCode=500 The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:05:43.561Z [qtp817348612-101] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "NS" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "e7a6d00a-9316-47af-81a2-1583055b2f35" INVARIANT_UUID = "78ff1a10-5eed-4eab-9611-ac77e6cb368e" PREV_VERSION = "1.0" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=919f9a12-dde7-4248-8957-a6b22734ef89 TargetServiceName=null EntryTimestamp=2021-07-13T00:05:30.017Z InvocationID=953eb0db-7c0d-47a9-b241-314873727cbe ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:05:44.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6681942-628b-4a90-843c-73fa4e685591 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:05:44.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6681942-628b-4a90-843c-73fa4e685591 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@ceda913 2021-07-13T00:05:44.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6681942-628b-4a90-843c-73fa4e685591 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:05:44.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6681942-628b-4a90-843c-73fa4e685591 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:05:47.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d9d18a3-ead9-4b2e-a505-fba8932b83d1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:05:47.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d9d18a3-ead9-4b2e-a505-fba8932b83d1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6f33fcf1 2021-07-13T00:05:47.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d9d18a3-ead9-4b2e-a505-fba8932b83d1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:05:47.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d9d18a3-ead9-4b2e-a505-fba8932b83d1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:05:48.961Z [qtp817348612-101] INFO o.o.s.b.m.o.impl.GraphLockOperation AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "NS" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "e7a6d00a-9316-47af-81a2-1583055b2f35" INVARIANT_UUID = "78ff1a10-5eed-4eab-9611-ac77e6cb368e" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" STATUS = "200" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.NS" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=919f9a12-dde7-4248-8957-a6b22734ef89 TargetServiceName=null EntryTimestamp=2021-07-13T00:05:30.017Z InvocationID=953eb0db-7c0d-47a9-b241-314873727cbe ErrorCode=500 lock resource with id 08ae43c2-911e-4d27-80c1-a9d79db6649f 2021-07-13T00:05:50.060Z [qtp817348612-101] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "NS" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "e7a6d00a-9316-47af-81a2-1583055b2f35" INVARIANT_UUID = "78ff1a10-5eed-4eab-9611-ac77e6cb368e" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" STATUS = "200" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.NS" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=919f9a12-dde7-4248-8957-a6b22734ef89 TargetServiceName=null EntryTimestamp=2021-07-13T00:05:30.017Z InvocationID=953eb0db-7c0d-47a9-b241-314873727cbe ErrorCode=500 owner of resource 08ae43c2-911e-4d27-80c1-a9d79db6649f is jh0003 2021-07-13T00:05:50.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dde5fb74-5a3d-4930-834c-5bf6bcfe74f0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:05:50.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dde5fb74-5a3d-4930-834c-5bf6bcfe74f0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1be826a4 2021-07-13T00:05:50.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dde5fb74-5a3d-4930-834c-5bf6bcfe74f0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:05:50.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dde5fb74-5a3d-4930-834c-5bf6bcfe74f0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:05:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:05:52.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:05:54.171Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89e0ddbd-7f20-4d15-90da-00b66dfc5fcf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:05:54.171Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89e0ddbd-7f20-4d15-90da-00b66dfc5fcf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@de66b1f 2021-07-13T00:05:54.171Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89e0ddbd-7f20-4d15-90da-00b66dfc5fcf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:05:54.171Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89e0ddbd-7f20-4d15-90da-00b66dfc5fcf PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:05:56.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4e7e316-dd14-40b7-bce7-30097cb0fb0b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:05:56.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4e7e316-dd14-40b7-bce7-30097cb0fb0b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6ca2fe81 2021-07-13T00:05:56.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4e7e316-dd14-40b7-bce7-30097cb0fb0b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:05:56.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4e7e316-dd14-40b7-bce7-30097cb0fb0b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:05:59.262Z [qtp817348612-101] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "NS" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "919f9a12-dde7-4248-8957-a6b22734ef89" INVARIANT_UUID = "78ff1a10-5eed-4eab-9611-ac77e6cb368e" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=919f9a12-dde7-4248-8957-a6b22734ef89 TargetServiceName=null EntryTimestamp=2021-07-13T00:05:30.017Z InvocationID=953eb0db-7c0d-47a9-b241-314873727cbe ErrorCode=500 owner of resource 08ae43c2-911e-4d27-80c1-a9d79db6649f is jh0003 2021-07-13T00:05:59.263Z [qtp817348612-101] INFO o.o.s.b.c.l.CertificationChangeTransition AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "NS" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "919f9a12-dde7-4248-8957-a6b22734ef89" INVARIANT_UUID = "78ff1a10-5eed-4eab-9611-ac77e6cb368e" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=919f9a12-dde7-4248-8957-a6b22734ef89 TargetServiceName=null EntryTimestamp=2021-07-13T00:05:30.017Z InvocationID=953eb0db-7c0d-47a9-b241-314873727cbe ErrorCode=500 validate before certification change. resource name=NS, oldState=NOT_CERTIFIED_CHECKIN, owner userId=jh0003 2021-07-13T00:05:59.263Z [qtp817348612-101] INFO o.o.s.b.c.l.CertificationChangeTransition AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "NS" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "919f9a12-dde7-4248-8957-a6b22734ef89" INVARIANT_UUID = "78ff1a10-5eed-4eab-9611-ac77e6cb368e" PREV_VERSION = "1.1" CURR_VERSION = "1.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=919f9a12-dde7-4248-8957-a6b22734ef89 TargetServiceName=null EntryTimestamp=2021-07-13T00:05:30.017Z InvocationID=953eb0db-7c0d-47a9-b241-314873727cbe ErrorCode=500 start performing certification change for resource 08ae43c2-911e-4d27-80c1-a9d79db6649f 2021-07-13T00:05:59.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50b550d6-6acb-49a9-812e-0fdc94f3ac5e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:05:59.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50b550d6-6acb-49a9-812e-0fdc94f3ac5e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c064390 2021-07-13T00:05:59.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50b550d6-6acb-49a9-812e-0fdc94f3ac5e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:05:59.871Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50b550d6-6acb-49a9-812e-0fdc94f3ac5e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:06:03.266Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=189930c9-2491-4d1b-83c5-e8504dbfd181 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:06:03.266Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=189930c9-2491-4d1b-83c5-e8504dbfd181 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@46018bb2 2021-07-13T00:06:03.266Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=189930c9-2491-4d1b-83c5-e8504dbfd181 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:06:03.266Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=189930c9-2491-4d1b-83c5-e8504dbfd181 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:06:05.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c69b5897-9472-4b4c-a48a-8d197127a2dd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:06:05.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c69b5897-9472-4b4c-a48a-8d197127a2dd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7881aefe 2021-07-13T00:06:05.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c69b5897-9472-4b4c-a48a-8d197127a2dd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:06:05.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c69b5897-9472-4b4c-a48a-8d197127a2dd PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:06:07.874Z [qtp817348612-101] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "NS" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "919f9a12-dde7-4248-8957-a6b22734ef89" INVARIANT_UUID = "78ff1a10-5eed-4eab-9611-ac77e6cb368e" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "a52a7305-3368-4f9b-85fb-e7adaf38df1e.assettoscatemplate" CURR_ARTIFACT_UUID = "a52a7305-3368-4f9b-85fb-e7adaf38df1e.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-Ns-template.yml,0,a52a7305-3368-4f9b-85fb-e7adaf38df1e.assettoscatemplate,1,931b2bd5-2793-4d52-b4c2-f63cf48b8a70" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=3100c382-011b-4103-af51-ec65c16986a9 TargetServiceName=null EntryTimestamp=2021-07-13T00:05:30.017Z InvocationID=953eb0db-7c0d-47a9-b241-314873727cbe ErrorCode=500 Starting to create Csar for component NS 2021-07-13T00:06:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:06:07.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:06:08.176Z [qtp817348612-101] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "NS" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "919f9a12-dde7-4248-8957-a6b22734ef89" INVARIANT_UUID = "78ff1a10-5eed-4eab-9611-ac77e6cb368e" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "a52a7305-3368-4f9b-85fb-e7adaf38df1e.assettoscatemplate" CURR_ARTIFACT_UUID = "a52a7305-3368-4f9b-85fb-e7adaf38df1e.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-Ns-template.yml,0,a52a7305-3368-4f9b-85fb-e7adaf38df1e.assettoscatemplate,1,931b2bd5-2793-4d52-b4c2-f63cf48b8a70" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=3100c382-011b-4103-af51-ec65c16986a9 TargetServiceName=null EntryTimestamp=2021-07-13T00:05:30.017Z InvocationID=953eb0db-7c0d-47a9-b241-314873727cbe ErrorCode=500 Ended create Csar for component NS 2021-07-13T00:06:09.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae8b2c85-b202-494f-8594-1db6339eddd0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:06:09.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae8b2c85-b202-494f-8594-1db6339eddd0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@32c0199 2021-07-13T00:06:09.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae8b2c85-b202-494f-8594-1db6339eddd0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:06:09.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae8b2c85-b202-494f-8594-1db6339eddd0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:06:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:06:09.367Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:06:09.367Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:06:11.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56cd72f5-7c75-44ff-adb0-a4a4b4917591 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:06:11.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56cd72f5-7c75-44ff-adb0-a4a4b4917591 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d98c8af 2021-07-13T00:06:11.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56cd72f5-7c75-44ff-adb0-a4a4b4917591 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:06:11.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=56cd72f5-7c75-44ff-adb0-a4a4b4917591 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:06:15.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdedec0d-2044-4005-9f70-7ea0b31d0d7c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:06:15.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdedec0d-2044-4005-9f70-7ea0b31d0d7c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@678e8ab5 2021-07-13T00:06:15.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdedec0d-2044-4005-9f70-7ea0b31d0d7c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:06:15.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdedec0d-2044-4005-9f70-7ea0b31d0d7c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:06:17.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3287fd69-840b-46da-b42e-5ca048c0ce85 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:06:17.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3287fd69-840b-46da-b42e-5ca048c0ce85 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@120d2ee4 2021-07-13T00:06:17.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3287fd69-840b-46da-b42e-5ca048c0ce85 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:06:17.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3287fd69-840b-46da-b42e-5ca048c0ce85 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:06:19.416Z [qtp817348612-101] INFO o.j.d.util.BackendOperation AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "NS" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "919f9a12-dde7-4248-8957-a6b22734ef89" INVARIANT_UUID = "78ff1a10-5eed-4eab-9611-ac77e6cb368e" PREV_VERSION = "" CURR_VERSION = "1.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "a52a7305-3368-4f9b-85fb-e7adaf38df1e.assettoscacsar" CURR_ARTIFACT_UUID = "a52a7305-3368-4f9b-85fb-e7adaf38df1e.assettoscacsar" ARTIFACT_DATA = "TOSCA,'assettoscacsar',TOSCA_CSAR,resource-Ns-csar.csar,0,a52a7305-3368-4f9b-85fb-e7adaf38df1e.assettoscacsar,1,c7a92943-8c9b-4800-9add-ab2a95418b96" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=0cb83a8f-500d-44e4-8314-865c3c0698b9 TargetServiceName=null EntryTimestamp=2021-07-13T00:05:30.017Z InvocationID=953eb0db-7c0d-47a9-b241-314873727cbe ErrorCode=500 Temporary exception during backend operation [CacheMutation]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateManyUnlogged(CQLStoreManager.java:504) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateMany(CQLStoreManager.java:435) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStoreManager.mutateMany(ExpectedValueCheckingStoreManager.java:76) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:94) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:91) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:91) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:139) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:196) at org.janusgraph.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:134) at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:733) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1379) at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.doCommit(JanusGraphBlueprintsGraph.java:298) at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:104) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.commit(JanusGraphClient.java:250) at org.openecomp.sdc.be.dao.jsongraph.JanusGraphDao.commit(JanusGraphDao.java:76) at org.openecomp.sdc.be.components.lifecycle.CertificationChangeTransition.changeState(CertificationChangeTransition.java:170) at org.openecomp.sdc.be.components.lifecycle.LifecycleBusinessLogic.changeState(LifecycleBusinessLogic.java:258) at org.openecomp.sdc.be.components.lifecycle.LifecycleBusinessLogic.changeComponentState(LifecycleBusinessLogic.java:191) at org.openecomp.sdc.be.components.lifecycle.LifecycleBusinessLogic.changeState(LifecycleBusinessLogic.java:128) at org.openecomp.sdc.be.components.impl.ResourceBusinessLogic.nodeFullCertification(ResourceBusinessLogic.java:3281) at org.openecomp.sdc.be.components.impl.ResourceBusinessLogic.propagateStateToCertified(ResourceBusinessLogic.java:3261) at org.openecomp.sdc.be.components.impl.ResourceImportManager.importCertifiedResource(ResourceImportManager.java:200) at org.openecomp.sdc.be.components.impl.ResourceImportManager.importNormativeResource(ResourceImportManager.java:143) at org.openecomp.sdc.be.servlets.AbstractValidationsServlet.handleImport(AbstractValidationsServlet.java:659) at org.openecomp.sdc.be.servlets.ResourceUploadServlet.uploadMultipart_aroundBody0(ResourceUploadServlet.java:131) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$AjcClosure1.run(ResourceUploadServlet.java:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:212) at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) at com.jcabi.aspects.aj.MethodLogger.wrapClass(MethodLogger.java:136) at org.openecomp.sdc.be.servlets.ResourceUploadServlet.uploadMultipart(ResourceUploadServlet.java:109) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$$FastClassBySpringCGLIB$$6732af97.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$$EnhancerBySpringCGLIB$$66c7ac05.uploadMultipart() at jdk.internal.reflect.GeneratedMethodAccessor368.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1411) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:226) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.togglz.servlet.TogglzFilter.doFilter(TogglzFilter.java:100) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.ReqValidationFilter.doFilter(ReqValidationFilter.java:72) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.GatewayFilter.doFilter(GatewayFilter.java:71) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1630) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:501) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:113) at com.datastax.driver.core.Responses$Error.asException(Responses.java:167) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:87) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:06:20.124Z [qtp817348612-101] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "NS" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "919f9a12-dde7-4248-8957-a6b22734ef89" INVARIANT_UUID = "78ff1a10-5eed-4eab-9611-ac77e6cb368e" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=0cb83a8f-500d-44e4-8314-865c3c0698b9 TargetServiceName=null EntryTimestamp=2021-07-13T00:05:30.017Z InvocationID=953eb0db-7c0d-47a9-b241-314873727cbe ErrorCode=500 The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:06:20.124Z [qtp817348612-101] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "NS" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "919f9a12-dde7-4248-8957-a6b22734ef89" INVARIANT_UUID = "78ff1a10-5eed-4eab-9611-ac77e6cb368e" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=0cb83a8f-500d-44e4-8314-865c3c0698b9 TargetServiceName=null EntryTimestamp=2021-07-13T00:05:30.017Z InvocationID=953eb0db-7c0d-47a9-b241-314873727cbe ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:06:20.124Z [qtp817348612-101] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "NS" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "919f9a12-dde7-4248-8957-a6b22734ef89" INVARIANT_UUID = "78ff1a10-5eed-4eab-9611-ac77e6cb368e" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=0cb83a8f-500d-44e4-8314-865c3c0698b9 TargetServiceName=null EntryTimestamp=2021-07-13T00:05:30.017Z InvocationID=953eb0db-7c0d-47a9-b241-314873727cbe ErrorCode=500 unlock component 08ae43c2-911e-4d27-80c1-a9d79db6649f 2021-07-13T00:06:20.124Z [qtp817348612-101] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "NS" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "919f9a12-dde7-4248-8957-a6b22734ef89" INVARIANT_UUID = "78ff1a10-5eed-4eab-9611-ac77e6cb368e" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=0cb83a8f-500d-44e4-8314-865c3c0698b9 TargetServiceName=null EntryTimestamp=2021-07-13T00:05:30.017Z InvocationID=953eb0db-7c0d-47a9-b241-314873727cbe ErrorCode=500 During change state, another component 08ae43c2-911e-4d27-80c1-a9d79db6649f has been created/updated 2021-07-13T00:06:21.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4e3bc87-94c0-4167-94d3-2d4327d59e2b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:06:21.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4e3bc87-94c0-4167-94d3-2d4327d59e2b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@71cec5f5 2021-07-13T00:06:21.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4e3bc87-94c0-4167-94d3-2d4327d59e2b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:06:21.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4e3bc87-94c0-4167-94d3-2d4327d59e2b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:06:21.442Z [qtp817348612-101] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "NS" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "919f9a12-dde7-4248-8957-a6b22734ef89" INVARIANT_UUID = "78ff1a10-5eed-4eab-9611-ac77e6cb368e" PREV_VERSION = "1.1" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR ResponseCode=201 RequestID=null ElapsedTime=51424 PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp StatusCode=COMPLETE TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=0cb83a8f-500d-44e4-8314-865c3c0698b9 TargetServiceName=null EntryTimestamp=2021-07-13T00:05:30.017Z InvocationID=953eb0db-7c0d-47a9-b241-314873727cbe ErrorCode=500 LogTimestamp=2021-07-13T00:06:21.441360Z ResponseDesc=Created Exiting. 2021-07-13T00:06:21.442Z [qtp817348612-101] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 SC="201" 2021-07-13T00:06:21.472Z [qtp817348612-17] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=81f08b6b-4831-4e71-805a-662c7f716046 InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS RemoteHost=10.242.77.79 ServiceName=v1/catalog/upload/multipart ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:06:21.471284Z InvocationID=99f48f2b-6331-499e-b5bb-6bc598557c66 LogTimestamp=2021-07-13T00:06:21.471323Z Entering 2021-07-13T00:06:21.472Z [qtp817348612-17] INFO o.o.sdc.be.filters.BeServletFilter RequestID=81f08b6b-4831-4e71-805a-662c7f716046 InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS userId=jh0003 RemoteHost=10.242.77.79 ServiceName=v1/catalog/upload/multipart ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:06:21.471284Z InvocationID=99f48f2b-6331-499e-b5bb-6bc598557c66 LogTimestamp=2021-07-13T00:06:21.471323Z localAddr=10.242.98.212 remoteAddr=10.242.77.79 POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 2021-07-13T00:06:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:06:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:06:23.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9adea4ef-3c27-45b6-b7a6-309e61013946 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:06:23.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9adea4ef-3c27-45b6-b7a6-309e61013946 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1af8b6d3 2021-07-13T00:06:23.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9adea4ef-3c27-45b6-b7a6-309e61013946 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:06:23.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9adea4ef-3c27-45b6-b7a6-309e61013946 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:06:24.822Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:06:26.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3935d766-19ce-4b8d-a232-05062e03f44c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:06:26.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3935d766-19ce-4b8d-a232-05062e03f44c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@9304c1b 2021-07-13T00:06:26.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3935d766-19ce-4b8d-a232-05062e03f44c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:06:26.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3935d766-19ce-4b8d-a232-05062e03f44c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:06:27.069Z [qtp817348612-17] INFO o.o.s.b.m.o.impl.GraphLockOperation RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:06:21.473Z InvocationID=8a1e47be-464b-4012-8623-95aaa356eda8 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp lock resource with id 6a1a5cee-acdb-4880-aa85-7a5c74cbad9a 2021-07-13T00:06:30.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=161bea8f-b07a-4619-808d-13ca360fa30c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:06:30.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=161bea8f-b07a-4619-808d-13ca360fa30c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@723139d7 2021-07-13T00:06:30.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=161bea8f-b07a-4619-808d-13ca360fa30c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:06:30.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=161bea8f-b07a-4619-808d-13ca360fa30c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:06:33.062Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=161bea8f-b07a-4619-808d-13ca360fa30c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:06:33.063Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=161bea8f-b07a-4619-808d-13ca360fa30c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:06:33.063Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=161bea8f-b07a-4619-808d-13ca360fa30c ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:06:33.063Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=161bea8f-b07a-4619-808d-13ca360fa30c ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:06:34.069Z [qtp817348612-17] INFO o.o.s.b.c.l.LifecycleBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:06:21.473Z InvocationID=8a1e47be-464b-4012-8623-95aaa356eda8 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp owner of resource 6a1a5cee-acdb-4880-aa85-7a5c74cbad9a is jh0003 2021-07-13T00:06:34.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00ac0d50-1bbe-43b2-88d4-7264cc60c14a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:06:34.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00ac0d50-1bbe-43b2-88d4-7264cc60c14a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7557bb09 2021-07-13T00:06:34.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00ac0d50-1bbe-43b2-88d4-7264cc60c14a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:06:34.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00ac0d50-1bbe-43b2-88d4-7264cc60c14a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:06:34.967Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=00ac0d50-1bbe-43b2-88d4-7264cc60c14a ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:06:34.967Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=00ac0d50-1bbe-43b2-88d4-7264cc60c14a ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:06:35.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7894edc3-4869-4ca6-becc-776a1f7a4ec1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:06:35.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7894edc3-4869-4ca6-becc-776a1f7a4ec1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@211d0489 2021-07-13T00:06:35.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7894edc3-4869-4ca6-becc-776a1f7a4ec1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:06:35.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7894edc3-4869-4ca6-becc-776a1f7a4ec1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:06:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:06:37.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:06:39.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e784424-fb07-4d33-ad8a-c1a32035c320 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:06:39.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e784424-fb07-4d33-ad8a-c1a32035c320 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5a196efc 2021-07-13T00:06:39.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e784424-fb07-4d33-ad8a-c1a32035c320 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:06:39.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e784424-fb07-4d33-ad8a-c1a32035c320 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:06:41.560Z [qtp817348612-17] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.1" CURR_VERSION = "1.2" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:06:21.473Z InvocationID=8a1e47be-464b-4012-8623-95aaa356eda8 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:06:41.561Z [qtp817348612-17] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.1" CURR_VERSION = "1.2" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:06:21.473Z InvocationID=8a1e47be-464b-4012-8623-95aaa356eda8 ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:06:41.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78d9cafb-1964-420a-994c-ae6379a1f1d6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:06:41.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78d9cafb-1964-420a-994c-ae6379a1f1d6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@12de1945 2021-07-13T00:06:41.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78d9cafb-1964-420a-994c-ae6379a1f1d6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:06:41.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78d9cafb-1964-420a-994c-ae6379a1f1d6 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:06:44.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3478e3e7-f2c7-4314-8515-00157240b5b4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:06:44.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3478e3e7-f2c7-4314-8515-00157240b5b4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e2ec275 2021-07-13T00:06:44.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3478e3e7-f2c7-4314-8515-00157240b5b4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:06:44.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3478e3e7-f2c7-4314-8515-00157240b5b4 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:06:47.670Z [qtp817348612-17] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.1" CURR_VERSION = "1.2" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:06:21.473Z InvocationID=8a1e47be-464b-4012-8623-95aaa356eda8 ErrorCode=500 The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:06:47.671Z [qtp817348612-17] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.1" CURR_VERSION = "1.2" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:06:21.473Z InvocationID=8a1e47be-464b-4012-8623-95aaa356eda8 ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:06:47.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=261a913a-7399-4e95-be86-a031883fb5d8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:06:47.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=261a913a-7399-4e95-be86-a031883fb5d8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@475eb675 2021-07-13T00:06:47.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=261a913a-7399-4e95-be86-a031883fb5d8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:06:47.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=261a913a-7399-4e95-be86-a031883fb5d8 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:06:50.968Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=261a913a-7399-4e95-be86-a031883fb5d8 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:06:50.969Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=261a913a-7399-4e95-be86-a031883fb5d8 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:06:50.969Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=261a913a-7399-4e95-be86-a031883fb5d8 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:06:50.969Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=261a913a-7399-4e95-be86-a031883fb5d8 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:06:51.524Z [qtp817348612-17] INFO o.j.d.util.BackendOperation AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.2" CURR_VERSION = "1.2" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" STATUS = "200" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.VNF" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:06:21.473Z InvocationID=8a1e47be-464b-4012-8623-95aaa356eda8 ErrorCode=500 Temporary exception during backend operation [CacheMutation]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateManyUnlogged(CQLStoreManager.java:504) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateMany(CQLStoreManager.java:435) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStoreManager.mutateMany(ExpectedValueCheckingStoreManager.java:76) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:94) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:91) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:91) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:139) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:196) at org.janusgraph.diskstorage.BackendTransaction.commit(BackendTransaction.java:151) at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:707) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1379) at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.doCommit(JanusGraphBlueprintsGraph.java:298) at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:104) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.commit(JanusGraphClient.java:250) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.unlockNode(JanusGraphGenericDao.java:1272) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.releaseElement(JanusGraphGenericDao.java:1284) at org.openecomp.sdc.be.model.operations.impl.GraphLockOperation.unlockComponent(GraphLockOperation.java:65) at org.openecomp.sdc.be.components.impl.ResourceBusinessLogic.updateExistingResourceByImport(ResourceBusinessLogic.java:3460) at org.openecomp.sdc.be.components.impl.ResourceBusinessLogic.createOrUpdateResourceByImport(ResourceBusinessLogic.java:3314) at org.openecomp.sdc.be.components.impl.ResourceImportManager.importCertifiedResource(ResourceImportManager.java:188) at org.openecomp.sdc.be.components.impl.ResourceImportManager.importNormativeResource(ResourceImportManager.java:143) at org.openecomp.sdc.be.servlets.AbstractValidationsServlet.handleImport(AbstractValidationsServlet.java:659) at org.openecomp.sdc.be.servlets.ResourceUploadServlet.uploadMultipart_aroundBody0(ResourceUploadServlet.java:131) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$AjcClosure1.run(ResourceUploadServlet.java:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:212) at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) at com.jcabi.aspects.aj.MethodLogger.wrapClass(MethodLogger.java:136) at org.openecomp.sdc.be.servlets.ResourceUploadServlet.uploadMultipart(ResourceUploadServlet.java:109) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$$FastClassBySpringCGLIB$$6732af97.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$$EnhancerBySpringCGLIB$$66c7ac05.uploadMultipart() at jdk.internal.reflect.GeneratedMethodAccessor368.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1411) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:226) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.togglz.servlet.TogglzFilter.doFilter(TogglzFilter.java:100) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.ReqValidationFilter.doFilter(ReqValidationFilter.java:72) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.GatewayFilter.doFilter(GatewayFilter.java:71) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1630) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:501) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:113) at com.datastax.driver.core.Responses$Error.asException(Responses.java:167) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:87) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:06:52.071Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00fbe379-068e-4d9f-858c-de2ce82638b4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:06:52.071Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00fbe379-068e-4d9f-858c-de2ce82638b4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@56a15eec 2021-07-13T00:06:52.071Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00fbe379-068e-4d9f-858c-de2ce82638b4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:06:52.071Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00fbe379-068e-4d9f-858c-de2ce82638b4 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:06:52.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:06:52.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:06:55.113Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=00fbe379-068e-4d9f-858c-de2ce82638b4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:06:55.114Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=00fbe379-068e-4d9f-858c-de2ce82638b4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:06:55.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:06:55.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:06:56.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d367327-35a8-4054-b312-a7f4eaedca9c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:06:56.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d367327-35a8-4054-b312-a7f4eaedca9c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4c1230d9 2021-07-13T00:06:56.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d367327-35a8-4054-b312-a7f4eaedca9c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:06:56.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8d367327-35a8-4054-b312-a7f4eaedca9c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:06:58.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:06:59.122Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=8d367327-35a8-4054-b312-a7f4eaedca9c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:06:59.122Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=8d367327-35a8-4054-b312-a7f4eaedca9c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:07:00.169Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3eecabf7-376f-4470-a251-c83eebcff6ac PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:07:00.169Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3eecabf7-376f-4470-a251-c83eebcff6ac PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@22c7bd2b 2021-07-13T00:07:00.169Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3eecabf7-376f-4470-a251-c83eebcff6ac PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:07:00.170Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3eecabf7-376f-4470-a251-c83eebcff6ac PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:07:01.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:07:03.229Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=3eecabf7-376f-4470-a251-c83eebcff6ac ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:07:03.229Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=3eecabf7-376f-4470-a251-c83eebcff6ac ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:07:03.229Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=3eecabf7-376f-4470-a251-c83eebcff6ac ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:07:03.229Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=3eecabf7-376f-4470-a251-c83eebcff6ac ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:07:04.263Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aaf6fbb9-7fd8-4af7-9000-7105c4d7b541 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:07:04.263Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aaf6fbb9-7fd8-4af7-9000-7105c4d7b541 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@705c9cbd 2021-07-13T00:07:04.263Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aaf6fbb9-7fd8-4af7-9000-7105c4d7b541 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:07:04.263Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aaf6fbb9-7fd8-4af7-9000-7105c4d7b541 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:07:04.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:07:05.415Z [qtp817348612-17] WARN o.j.d.l.c.ConsistentKeyLocker AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.2" CURR_VERSION = "1.2" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" STATUS = "200" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.VNF" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:06:21.473Z InvocationID=8a1e47be-464b-4012-8623-95aaa356eda8 ErrorCode=500 Lock write succeeded but took too long: duration PT5.952161S exceeded limit PT0.5S 2021-07-13T00:07:05.418Z [JanusGraph-Health-Check-Thread] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [VertexIndexQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:399) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:396) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.indexQuery(BackendTransaction.java:396) at org.janusgraph.graphdb.query.graph.MultiKeySliceQuery.execute(MultiKeySliceQuery.java:51) at org.janusgraph.graphdb.database.IndexSerializer.query(IndexSerializer.java:512) at org.janusgraph.graphdb.util.SubqueryIterator.(SubqueryIterator.java:66) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1295) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1190) at org.janusgraph.graphdb.query.QueryProcessor$LimitAdjustingIterator.getNewIterator(QueryProcessor.java:194) at org.janusgraph.graphdb.query.LimitAdjustingIterator.hasNext(LimitAdjustingIterator.java:68) at org.janusgraph.graphdb.query.ResultSetIterator.nextInternal(ResultSetIterator.java:54) at org.janusgraph.graphdb.query.ResultSetIterator.(ResultSetIterator.java:44) at org.janusgraph.graphdb.query.QueryProcessor.iterator(QueryProcessor.java:66) at com.google.common.collect.Iterables$4.iterator(Iterables.java:559) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:340) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:336) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ... 4 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:308) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:422) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 26 common frames omitted 2021-07-13T00:07:06.263Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=aaf6fbb9-7fd8-4af7-9000-7105c4d7b541 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:07:06.263Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=aaf6fbb9-7fd8-4af7-9000-7105c4d7b541 ErrorCategory=ERROR ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:07:06.815Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:07:07.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=157eabed-da98-4a75-bb18-6ed60666591c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:07:07.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=157eabed-da98-4a75-bb18-6ed60666591c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7fe11318 2021-07-13T00:07:07.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=157eabed-da98-4a75-bb18-6ed60666591c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:07:07.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=157eabed-da98-4a75-bb18-6ed60666591c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:07:07.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:07:07.933Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:07:08.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d44d8c4c-7122-4261-8f57-dc86c2807969 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:07:08.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d44d8c4c-7122-4261-8f57-dc86c2807969 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7ddf25cf 2021-07-13T00:07:08.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d44d8c4c-7122-4261-8f57-dc86c2807969 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:07:08.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d44d8c4c-7122-4261-8f57-dc86c2807969 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:07:09.159Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=d44d8c4c-7122-4261-8f57-dc86c2807969 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:07:09.160Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=d44d8c4c-7122-4261-8f57-dc86c2807969 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:07:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:07:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:07:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:07:10.169Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ecae0a68-56e1-4ca3-97a8-fb9bbcc967cf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:07:10.169Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ecae0a68-56e1-4ca3-97a8-fb9bbcc967cf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@69f2f7bd 2021-07-13T00:07:10.169Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ecae0a68-56e1-4ca3-97a8-fb9bbcc967cf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:07:10.169Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ecae0a68-56e1-4ca3-97a8-fb9bbcc967cf PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:07:11.267Z [qtp817348612-17] INFO o.o.s.b.m.o.impl.GraphLockOperation AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.2" CURR_VERSION = "1.2" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" STATUS = "200" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.VNF" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:06:21.473Z InvocationID=8a1e47be-464b-4012-8623-95aaa356eda8 ErrorCode=500 lock resource with id 5b01ce64-acd1-432b-bb05-a67a71fb6e71 2021-07-13T00:07:11.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=314d7010-bd58-4620-9f1d-43be4aa7e585 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:07:11.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=314d7010-bd58-4620-9f1d-43be4aa7e585 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@f62a8eb 2021-07-13T00:07:11.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=314d7010-bd58-4620-9f1d-43be4aa7e585 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:07:11.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=314d7010-bd58-4620-9f1d-43be4aa7e585 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:07:12.082Z [qtp817348612-17] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.2" CURR_VERSION = "1.2" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" STATUS = "200" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.VNF" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:06:21.473Z InvocationID=8a1e47be-464b-4012-8623-95aaa356eda8 ErrorCode=500 owner of resource 5b01ce64-acd1-432b-bb05-a67a71fb6e71 is jh0003 2021-07-13T00:07:14.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1518e92-e942-440a-92af-0906867c70b9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:07:14.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1518e92-e942-440a-92af-0906867c70b9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78ec03cc 2021-07-13T00:07:14.974Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1518e92-e942-440a-92af-0906867c70b9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:07:14.974Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b1518e92-e942-440a-92af-0906867c70b9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:07:18.363Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8ecdf0c-eca5-47c9-a76e-5b25a1d6ae59 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:07:18.363Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8ecdf0c-eca5-47c9-a76e-5b25a1d6ae59 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7c19b043 2021-07-13T00:07:18.363Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8ecdf0c-eca5-47c9-a76e-5b25a1d6ae59 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:07:18.363Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8ecdf0c-eca5-47c9-a76e-5b25a1d6ae59 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:07:20.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=353bd727-cc40-4e08-83e1-e9b78c08a233 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:07:20.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=353bd727-cc40-4e08-83e1-e9b78c08a233 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2a8ff62 2021-07-13T00:07:20.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=353bd727-cc40-4e08-83e1-e9b78c08a233 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:07:20.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=353bd727-cc40-4e08-83e1-e9b78c08a233 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:07:20.970Z [qtp817348612-17] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.2" CURR_VERSION = "1.2" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:06:21.473Z InvocationID=8a1e47be-464b-4012-8623-95aaa356eda8 ErrorCode=500 owner of resource 5b01ce64-acd1-432b-bb05-a67a71fb6e71 is jh0003 2021-07-13T00:07:20.970Z [qtp817348612-17] INFO o.o.s.b.c.l.CertificationChangeTransition AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.2" CURR_VERSION = "1.2" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:06:21.473Z InvocationID=8a1e47be-464b-4012-8623-95aaa356eda8 ErrorCode=500 validate before certification change. resource name=VNF, oldState=NOT_CERTIFIED_CHECKIN, owner userId=jh0003 2021-07-13T00:07:20.970Z [qtp817348612-17] INFO o.o.s.b.c.l.CertificationChangeTransition AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.2" CURR_VERSION = "1.2" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:06:21.473Z InvocationID=8a1e47be-464b-4012-8623-95aaa356eda8 ErrorCode=500 start performing certification change for resource 5b01ce64-acd1-432b-bb05-a67a71fb6e71 2021-07-13T00:07:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:07:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:07:23.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09b51fc3-9a8d-4f22-aa84-15020576a779 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:07:23.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09b51fc3-9a8d-4f22-aa84-15020576a779 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@18972a2c 2021-07-13T00:07:23.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09b51fc3-9a8d-4f22-aa84-15020576a779 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:07:23.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09b51fc3-9a8d-4f22-aa84-15020576a779 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:07:24.321Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:07:26.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfec8643-916f-4579-8875-1126a77f401a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:07:26.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfec8643-916f-4579-8875-1126a77f401a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@fdb891 2021-07-13T00:07:26.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfec8643-916f-4579-8875-1126a77f401a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:07:26.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfec8643-916f-4579-8875-1126a77f401a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:07:27.466Z [qtp817348612-17] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "" CURR_VERSION = "1.2" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate" CURR_ARTIFACT_UUID = "2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-Vnf-template.yml,0,2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate,1,1e0a017e-4724-4a07-aebd-f25962197db9" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=4aa81812-4011-4341-8af8-e452041831fb TargetServiceName=null EntryTimestamp=2021-07-13T00:06:21.473Z InvocationID=8a1e47be-464b-4012-8623-95aaa356eda8 ErrorCode=500 Starting to create Csar for component VNF 2021-07-13T00:07:27.716Z [qtp817348612-17] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "" CURR_VERSION = "1.2" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate" CURR_ARTIFACT_UUID = "2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-Vnf-template.yml,0,2a1c6b8e-024a-4f8c-8456-8cef2de565e0.assettoscatemplate,1,1e0a017e-4724-4a07-aebd-f25962197db9" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=4aa81812-4011-4341-8af8-e452041831fb TargetServiceName=null EntryTimestamp=2021-07-13T00:06:21.473Z InvocationID=8a1e47be-464b-4012-8623-95aaa356eda8 ErrorCode=500 Ended create Csar for component VNF 2021-07-13T00:07:30.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96851e77-4707-406a-b8fc-6cc61df33c5f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:07:30.467Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96851e77-4707-406a-b8fc-6cc61df33c5f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@31579b65 2021-07-13T00:07:30.467Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96851e77-4707-406a-b8fc-6cc61df33c5f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:07:30.467Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96851e77-4707-406a-b8fc-6cc61df33c5f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:07:32.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b7119fc5-8889-44de-be79-22a360461761 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:07:32.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b7119fc5-8889-44de-be79-22a360461761 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@744131c2 2021-07-13T00:07:32.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b7119fc5-8889-44de-be79-22a360461761 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:07:32.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b7119fc5-8889-44de-be79-22a360461761 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:07:34.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:07:34.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:07:35.765Z [qtp817348612-17] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.2" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=51a3eb62-1794-415a-b938-647825f904f1 TargetServiceName=null EntryTimestamp=2021-07-13T00:06:21.473Z InvocationID=8a1e47be-464b-4012-8623-95aaa356eda8 ErrorCode=500 The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:07:35.766Z [qtp817348612-17] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.2" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=51a3eb62-1794-415a-b938-647825f904f1 TargetServiceName=null EntryTimestamp=2021-07-13T00:06:21.473Z InvocationID=8a1e47be-464b-4012-8623-95aaa356eda8 ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:07:35.766Z [qtp817348612-17] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.2" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=51a3eb62-1794-415a-b938-647825f904f1 TargetServiceName=null EntryTimestamp=2021-07-13T00:06:21.473Z InvocationID=8a1e47be-464b-4012-8623-95aaa356eda8 ErrorCode=500 unlock component 5b01ce64-acd1-432b-bb05-a67a71fb6e71 2021-07-13T00:07:35.766Z [qtp817348612-17] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.2" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=51a3eb62-1794-415a-b938-647825f904f1 TargetServiceName=null EntryTimestamp=2021-07-13T00:06:21.473Z InvocationID=8a1e47be-464b-4012-8623-95aaa356eda8 ErrorCode=500 During change state, another component 5b01ce64-acd1-432b-bb05-a67a71fb6e71 has been created/updated 2021-07-13T00:07:35.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=addda445-de6d-4478-95bd-78b5c0ebefdb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:07:35.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=addda445-de6d-4478-95bd-78b5c0ebefdb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5526a32c 2021-07-13T00:07:35.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=addda445-de6d-4478-95bd-78b5c0ebefdb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:07:35.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=addda445-de6d-4478-95bd-78b5c0ebefdb PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:07:37.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:07:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:07:37.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:07:37.960Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=addda445-de6d-4478-95bd-78b5c0ebefdb ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:07:37.960Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=addda445-de6d-4478-95bd-78b5c0ebefdb ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:07:38.433Z [qtp817348612-17] WARN o.o.s.b.s.ResourceUploadServlet AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.2" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=51a3eb62-1794-415a-b938-647825f904f1 TargetServiceName=null EntryTimestamp=2021-07-13T00:06:21.473Z InvocationID=8a1e47be-464b-4012-8623-95aaa356eda8 ErrorCode=500 #uploadMultipart('multipart', /tmp/rep1662116913252952428tmp, 'form-data; filename="VNF.zip"; name="resourceZip"', '{"payloadName": "VNF.yml", "contactId": "jh0003", "name": "VNF", "description": "VNF", "resourceIconPath": "compute", "resourceType": "VFC", "categories": [{"name": "Generic", "subcategories": [{"name": "Infrastructure"}]}], "vendorName": "ETSI", "vendorRelease": "4.1.1", "tags": ["VNF"]}', 'Request(POST https://sdc-be.onap:8443/sdc2/rest/v1/catalog/upload/multipart?createNewVersion=true)@62228854', 'jh0003', true): org.apache.cxf.jaxrs.impl.ResponseImpl@444d1a88 in 1.28min (too slow!) 2021-07-13T00:07:38.434Z [qtp817348612-17] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VNF" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "10b1b941-7f72-4723-ac16-250117ebd8e3" INVARIANT_UUID = "2b09582d-6bd3-4aa8-a3c1-490c113aa8e1" PREV_VERSION = "1.2" CURR_VERSION = "2.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR ResponseCode=201 RequestID=null ElapsedTime=76960 PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp StatusCode=COMPLETE TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=51a3eb62-1794-415a-b938-647825f904f1 TargetServiceName=null EntryTimestamp=2021-07-13T00:06:21.473Z InvocationID=8a1e47be-464b-4012-8623-95aaa356eda8 ErrorCode=500 LogTimestamp=2021-07-13T00:07:38.433551Z ResponseDesc=Created Exiting. 2021-07-13T00:07:38.434Z [qtp817348612-17] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 SC="201" 2021-07-13T00:07:38.465Z [qtp817348612-96] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=8290f46d-7272-4867-993f-bd51ef9d96af InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS RemoteHost=10.242.77.79 ServiceName=v1/catalog/upload/multipart ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:07:38.465236Z InvocationID=1e7a23e4-054d-4765-8656-2f195e525253 LogTimestamp=2021-07-13T00:07:38.465288Z Entering 2021-07-13T00:07:38.465Z [qtp817348612-96] INFO o.o.sdc.be.filters.BeServletFilter RequestID=8290f46d-7272-4867-993f-bd51ef9d96af InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS userId=jh0003 RemoteHost=10.242.77.79 ServiceName=v1/catalog/upload/multipart ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:07:38.465236Z InvocationID=1e7a23e4-054d-4765-8656-2f195e525253 LogTimestamp=2021-07-13T00:07:38.465288Z localAddr=10.242.98.212 remoteAddr=10.242.77.79 POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 2021-07-13T00:07:38.466Z [qtp817348612-96] ERROR o.o.s.b.f.ComponentsAvailabilityFilter RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:07:38.465Z InvocationID=5cfa715b-d796-4981-b12f-f08dd3d51690 ErrorCode=500 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null Components Availability Filter Failed - ES/Cassandra is DOWN 2021-07-13T00:07:38.479Z [qtp817348612-96] ERROR o.o.s.b.f.ComponentsAvailabilityFilter ResponseCode=500 RequestID=null InstanceID=null ElapsedTime=1 EndTimestamp=2021-07-13T00:07:38.466Z Severity=0 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp StatusCode=ERROR TargetEntity=null RemoteHost=10.242.77.79 ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=null TargetServiceName=null EntryTimestamp=2021-07-13T00:07:38.465Z ClassName=org.openecomp.sdc.be.filters.ComponentsAvailabilityFilter InvocationID=5cfa715b-d796-4981-b12f-f08dd3d51690 ErrorCode=500 ResponseDesc=Internal Server Error org.openecomp.sdc.exception.ResponseFormat$RequestErrorWrapper@15ca891a 2021-07-13T00:07:38.479Z [qtp817348612-96] ERROR o.o.s.b.f.ComponentsAvailabilityFilter ResponseCode=500 RequestID=null InstanceID=null ElapsedTime=1 EndTimestamp=2021-07-13T00:07:38.466Z Severity=0 PartnerName=jh0003 auditOn=false ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp StatusCode=ERROR TargetEntity=null RemoteHost=10.242.77.79 ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=null TargetServiceName=null EntryTimestamp=2021-07-13T00:07:38.465Z ClassName=org.openecomp.sdc.be.filters.ComponentsAvailabilityFilter InvocationID=5cfa715b-d796-4981-b12f-f08dd3d51690 ErrorCode=500 ResponseDesc=Internal Server Error org.openecomp.sdc.exception.ResponseFormat$RequestErrorWrapper@15ca891a 2021-07-13T00:07:38.480Z [qtp817348612-96] WARN o.onap.logging.filter.base.MDCSetup ResponseCode=500 ServerIPAddress=10.242.98.212 ErrorDesc=Internal Server Error ErrorCode=500 LogTimestamp=2021-07-13T00:07:38.480177Z ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp StatusCode=ERROR ResponseDesc=Internal Server Error Unable to calculate elapsed time due to error: text 2021-07-13T00:07:38.480Z [qtp817348612-96] INFO o.o.l.f.base.AbstractAuditLogFilter ResponseCode=500 ServerIPAddress=10.242.98.212 ErrorDesc=Internal Server Error ErrorCode=500 LogTimestamp=2021-07-13T00:07:38.480177Z ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp StatusCode=ERROR ResponseDesc=Internal Server Error Exiting. 2021-07-13T00:07:38.480Z [qtp817348612-96] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 SC="500" 2021-07-13T00:07:39.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=279faf51-5e32-4632-96c3-08a9aaa0e20b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:07:39.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=279faf51-5e32-4632-96c3-08a9aaa0e20b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3872a029 2021-07-13T00:07:39.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=279faf51-5e32-4632-96c3-08a9aaa0e20b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:07:39.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=279faf51-5e32-4632-96c3-08a9aaa0e20b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:07:39.660Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:07:41.672Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cf79f52-8a80-4db5-95a8-402e260e7d77 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:07:41.672Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cf79f52-8a80-4db5-95a8-402e260e7d77 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@35f398f1 2021-07-13T00:07:41.672Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cf79f52-8a80-4db5-95a8-402e260e7d77 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:07:41.672Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5cf79f52-8a80-4db5-95a8-402e260e7d77 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:07:42.324Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=5cf79f52-8a80-4db5-95a8-402e260e7d77 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:07:42.324Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=5cf79f52-8a80-4db5-95a8-402e260e7d77 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:07:45.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bccfd2cb-4356-4d0f-80b7-e8d12151c8c9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:07:45.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bccfd2cb-4356-4d0f-80b7-e8d12151c8c9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@48c170ac 2021-07-13T00:07:45.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bccfd2cb-4356-4d0f-80b7-e8d12151c8c9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:07:45.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bccfd2cb-4356-4d0f-80b7-e8d12151c8c9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:07:47.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d79d85e-6c5d-40ce-ad0c-48f89b72c135 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:07:47.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d79d85e-6c5d-40ce-ad0c-48f89b72c135 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@289e955b 2021-07-13T00:07:47.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d79d85e-6c5d-40ce-ad0c-48f89b72c135 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:07:47.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d79d85e-6c5d-40ce-ad0c-48f89b72c135 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:07:48.720Z [qtp817348612-105] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=ce381cf7-752c-4de3-bf80-db1572faaa96 InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS RemoteHost=10.242.77.79 ServerIPAddress=10.242.98.212 ServiceName=v1/catalog/upload/multipart ServiceInstanceID=null EntryTimestamp=2021-07-13T00:07:48.719906Z InvocationID=64dc49af-96a9-4623-b0de-df1d9d691472 LogTimestamp=2021-07-13T00:07:48.720081Z Entering 2021-07-13T00:07:48.720Z [qtp817348612-105] INFO o.o.sdc.be.filters.BeServletFilter RequestID=ce381cf7-752c-4de3-bf80-db1572faaa96 InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS userId=jh0003 RemoteHost=10.242.77.79 ServerIPAddress=10.242.98.212 ServiceName=v1/catalog/upload/multipart ServiceInstanceID=null EntryTimestamp=2021-07-13T00:07:48.719906Z InvocationID=64dc49af-96a9-4623-b0de-df1d9d691472 LogTimestamp=2021-07-13T00:07:48.720081Z localAddr=10.242.98.212 remoteAddr=10.242.77.79 POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 2021-07-13T00:07:50.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71a619e4-e343-412a-a131-25d4215f6400 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:07:50.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71a619e4-e343-412a-a131-25d4215f6400 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5401b89 2021-07-13T00:07:50.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71a619e4-e343-412a-a131-25d4215f6400 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:07:50.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71a619e4-e343-412a-a131-25d4215f6400 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:07:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:07:52.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:07:53.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2de0e74-1c39-43ac-9b69-3139fe150f71 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:07:53.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2de0e74-1c39-43ac-9b69-3139fe150f71 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2cdd3fc0 2021-07-13T00:07:53.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2de0e74-1c39-43ac-9b69-3139fe150f71 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:07:53.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2de0e74-1c39-43ac-9b69-3139fe150f71 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:07:54.069Z [qtp817348612-105] INFO o.o.s.b.c.i.ComponentBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:07:48.721Z InvocationID=1308780b-35b9-4fab-8bdc-d2b60544af1b PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp LifecycleState cannot be defined by user. This field will be overridden by the application 2021-07-13T00:07:54.069Z [qtp817348612-105] INFO o.o.s.b.c.i.ComponentBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:07:48.721Z InvocationID=1308780b-35b9-4fab-8bdc-d2b60544af1b PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Version cannot be defined by user. This field will be overridden by the application 2021-07-13T00:07:54.069Z [qtp817348612-105] INFO o.o.s.b.c.i.ComponentBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:07:48.721Z InvocationID=1308780b-35b9-4fab-8bdc-d2b60544af1b PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Creator cannot be defined by user. This field will be overridden by the application 2021-07-13T00:07:54.069Z [qtp817348612-105] INFO o.o.s.b.c.i.ComponentBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:07:48.721Z InvocationID=1308780b-35b9-4fab-8bdc-d2b60544af1b PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Is Highest Version cannot be defined by user. This field will be overridden by the application 2021-07-13T00:07:54.159Z [qtp817348612-105] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Create Resource - validateCapabilityTypesCreate ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:07:48.721Z InvocationID=1308780b-35b9-4fab-8bdc-d2b60544af1b ErrorCode=400 Capability Type tosca.capabilities.nfv.ContainerGroupable required is missing 2021-07-13T00:07:54.159Z [qtp817348612-105] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Create Resource - validateCapabilityTypesCreate ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:07:48.721Z InvocationID=1308780b-35b9-4fab-8bdc-d2b60544af1b ErrorCode=500 Operation towards database failed. 2021-07-13T00:07:54.164Z [qtp817348612-105] INFO o.o.s.b.m.o.impl.GraphLockOperation AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "osContainer" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "" INVARIANT_UUID = "" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "" CURR_STATE = "CERTIFIED" STATUS = "400" DESC = "SVC4115: Error: Invalid Content. Missing Capability Type 'tosca.capabilities.nfv.ContainerGroupable'." TOSCA_NODE_TYPE = "tosca.nodes.nfv.Vdu.osContainer" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Create Resource - validateCapabilityTypesCreate ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:07:48.721Z InvocationID=1308780b-35b9-4fab-8bdc-d2b60544af1b ErrorCode=500 lock resource with name Oscontainer 2021-07-13T00:07:57.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e79c8dc6-0245-4f3c-ab8c-0b5e2363c5ed PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:07:57.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e79c8dc6-0245-4f3c-ab8c-0b5e2363c5ed PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6f5a7ee7 2021-07-13T00:07:57.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e79c8dc6-0245-4f3c-ab8c-0b5e2363c5ed PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:07:57.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e79c8dc6-0245-4f3c-ab8c-0b5e2363c5ed PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:08:00.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28db2012-2cea-4c9f-a649-05e5319dfb45 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:08:00.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28db2012-2cea-4c9f-a649-05e5319dfb45 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@33320a1a 2021-07-13T00:08:00.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28db2012-2cea-4c9f-a649-05e5319dfb45 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:08:00.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28db2012-2cea-4c9f-a649-05e5319dfb45 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:08:00.572Z [qtp817348612-105] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "osContainer" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "80199208-40e2-4d92-b05b-2254c40f889e" INVARIANT_UUID = "cb17aa3e-0d33-45e1-afa3-4399c191ab59" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "" CURR_STATE = "CERTIFIED" STATUS = "201" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.Vdu.osContainer" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Create Resource - validateCapabilityTypesCreate ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=80199208-40e2-4d92-b05b-2254c40f889e TargetServiceName=null EntryTimestamp=2021-07-13T00:07:48.721Z InvocationID=1308780b-35b9-4fab-8bdc-d2b60544af1b ErrorCode=500 The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:08:00.573Z [qtp817348612-105] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "osContainer" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "80199208-40e2-4d92-b05b-2254c40f889e" INVARIANT_UUID = "cb17aa3e-0d33-45e1-afa3-4399c191ab59" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "" CURR_STATE = "CERTIFIED" STATUS = "201" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.Vdu.osContainer" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=80199208-40e2-4d92-b05b-2254c40f889e TargetServiceName=null EntryTimestamp=2021-07-13T00:07:48.721Z InvocationID=1308780b-35b9-4fab-8bdc-d2b60544af1b ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:08:02.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94f6239f-d7f2-4281-92fe-66a825569542 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:08:02.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94f6239f-d7f2-4281-92fe-66a825569542 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@36abc3a6 2021-07-13T00:08:02.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94f6239f-d7f2-4281-92fe-66a825569542 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:08:02.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94f6239f-d7f2-4281-92fe-66a825569542 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:08:05.365Z [qtp817348612-105] INFO o.o.s.b.m.o.impl.GraphLockOperation AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "osContainer" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "80199208-40e2-4d92-b05b-2254c40f889e" INVARIANT_UUID = "cb17aa3e-0d33-45e1-afa3-4399c191ab59" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "" CURR_STATE = "CERTIFIED" STATUS = "201" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.Vdu.osContainer" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=80199208-40e2-4d92-b05b-2254c40f889e TargetServiceName=null EntryTimestamp=2021-07-13T00:07:48.721Z InvocationID=1308780b-35b9-4fab-8bdc-d2b60544af1b ErrorCode=500 lock resource with id 842eb6cf-72a2-4589-938a-912c51d7d1e6 2021-07-13T00:08:06.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=150976fd-ce9f-4078-8a1b-5ded3b19149f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:08:06.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=150976fd-ce9f-4078-8a1b-5ded3b19149f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@73cdd98c 2021-07-13T00:08:06.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=150976fd-ce9f-4078-8a1b-5ded3b19149f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:08:06.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=150976fd-ce9f-4078-8a1b-5ded3b19149f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:08:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:08:07.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:08:09.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e66d7b2-5aa3-4db2-be2e-0800da19fbfb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:08:09.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e66d7b2-5aa3-4db2-be2e-0800da19fbfb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4fdc3c54 2021-07-13T00:08:09.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e66d7b2-5aa3-4db2-be2e-0800da19fbfb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:08:09.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e66d7b2-5aa3-4db2-be2e-0800da19fbfb PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:08:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:08:09.367Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:08:09.367Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:08:10.111Z [qtp817348612-105] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "osContainer" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "80199208-40e2-4d92-b05b-2254c40f889e" INVARIANT_UUID = "cb17aa3e-0d33-45e1-afa3-4399c191ab59" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "842eb6cf-72a2-4589-938a-912c51d7d1e6.assettoscatemplate" CURR_ARTIFACT_UUID = "842eb6cf-72a2-4589-938a-912c51d7d1e6.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-Oscontainer-template.yml,0,842eb6cf-72a2-4589-938a-912c51d7d1e6.assettoscatemplate,0 " STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=64c9c03c-3358-4acd-96d3-584342887f86 TargetServiceName=null EntryTimestamp=2021-07-13T00:07:48.721Z InvocationID=1308780b-35b9-4fab-8bdc-d2b60544af1b ErrorCode=500 Starting to create Csar for component osContainer 2021-07-13T00:08:10.372Z [qtp817348612-105] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "osContainer" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "80199208-40e2-4d92-b05b-2254c40f889e" INVARIANT_UUID = "cb17aa3e-0d33-45e1-afa3-4399c191ab59" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "842eb6cf-72a2-4589-938a-912c51d7d1e6.assettoscatemplate" CURR_ARTIFACT_UUID = "842eb6cf-72a2-4589-938a-912c51d7d1e6.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-Oscontainer-template.yml,0,842eb6cf-72a2-4589-938a-912c51d7d1e6.assettoscatemplate,0 " STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=64c9c03c-3358-4acd-96d3-584342887f86 TargetServiceName=null EntryTimestamp=2021-07-13T00:07:48.721Z InvocationID=1308780b-35b9-4fab-8bdc-d2b60544af1b ErrorCode=500 Ended create Csar for component osContainer 2021-07-13T00:08:10.372Z [qtp817348612-105] INFO o.o.s.b.m.o.impl.GraphLockOperation AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "osContainer" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "80199208-40e2-4d92-b05b-2254c40f889e" INVARIANT_UUID = "cb17aa3e-0d33-45e1-afa3-4399c191ab59" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "842eb6cf-72a2-4589-938a-912c51d7d1e6.assettoscatemplate" CURR_ARTIFACT_UUID = "842eb6cf-72a2-4589-938a-912c51d7d1e6.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-Oscontainer-template.yml,0,842eb6cf-72a2-4589-938a-912c51d7d1e6.assettoscatemplate,0 " STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=64c9c03c-3358-4acd-96d3-584342887f86 TargetServiceName=null EntryTimestamp=2021-07-13T00:07:48.721Z InvocationID=1308780b-35b9-4fab-8bdc-d2b60544af1b ErrorCode=500 lock resource with id 842eb6cf-72a2-4589-938a-912c51d7d1e6 2021-07-13T00:08:11.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6529ee4-1460-44d6-85eb-70e5860219dd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:08:11.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6529ee4-1460-44d6-85eb-70e5860219dd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6c1d9d6f 2021-07-13T00:08:11.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6529ee4-1460-44d6-85eb-70e5860219dd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:08:11.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6529ee4-1460-44d6-85eb-70e5860219dd PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:08:14.169Z [qtp817348612-105] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "osContainer" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "80199208-40e2-4d92-b05b-2254c40f889e" INVARIANT_UUID = "cb17aa3e-0d33-45e1-afa3-4399c191ab59" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "842eb6cf-72a2-4589-938a-912c51d7d1e6.assettoscacsar" CURR_ARTIFACT_UUID = "842eb6cf-72a2-4589-938a-912c51d7d1e6.assettoscacsar" ARTIFACT_DATA = "TOSCA,'assettoscacsar',TOSCA_CSAR,resource-Oscontainer-csar.csar,0,842eb6cf-72a2-4589-938a-912c51d7d1e6.assettoscacsar,0 " STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR ResponseCode=201 RequestID=null ElapsedTime=25447 PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp StatusCode=COMPLETE TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=6fac83b0-b9bc-4b34-aa4a-2a11560bb966 TargetServiceName=null EntryTimestamp=2021-07-13T00:07:48.721Z InvocationID=1308780b-35b9-4fab-8bdc-d2b60544af1b ErrorCode=500 LogTimestamp=2021-07-13T00:08:14.168785Z ResponseDesc=Created Exiting. 2021-07-13T00:08:14.169Z [qtp817348612-105] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 SC="201" 2021-07-13T00:08:14.273Z [qtp817348612-18] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=e9e87140-446e-42a9-9c10-28dab8ba0cbd InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS RemoteHost=10.242.77.79 ServerIPAddress=10.242.98.212 ServiceName=v1/catalog/upload/multipart ServiceInstanceID=null EntryTimestamp=2021-07-13T00:08:14.273313Z InvocationID=d35e7b83-119c-4300-9253-162e3e583350 LogTimestamp=2021-07-13T00:08:14.273370Z Entering 2021-07-13T00:08:14.273Z [qtp817348612-18] INFO o.o.sdc.be.filters.BeServletFilter RequestID=e9e87140-446e-42a9-9c10-28dab8ba0cbd InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS userId=jh0003 RemoteHost=10.242.77.79 ServerIPAddress=10.242.98.212 ServiceName=v1/catalog/upload/multipart ServiceInstanceID=null EntryTimestamp=2021-07-13T00:08:14.273313Z InvocationID=d35e7b83-119c-4300-9253-162e3e583350 LogTimestamp=2021-07-13T00:08:14.273370Z localAddr=10.242.98.212 remoteAddr=10.242.77.79 POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 2021-07-13T00:08:14.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c0d787f-c4ae-4922-a044-efa203fd46a3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:08:14.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c0d787f-c4ae-4922-a044-efa203fd46a3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4c3d5b9 2021-07-13T00:08:14.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c0d787f-c4ae-4922-a044-efa203fd46a3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:08:14.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c0d787f-c4ae-4922-a044-efa203fd46a3 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:08:18.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0e58a18-a20e-4bc7-a527-1c0eed5288ba PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:08:18.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0e58a18-a20e-4bc7-a527-1c0eed5288ba PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@19df6d14 2021-07-13T00:08:18.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0e58a18-a20e-4bc7-a527-1c0eed5288ba PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:08:18.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0e58a18-a20e-4bc7-a527-1c0eed5288ba PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:08:21.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37d831d7-674c-4fc3-9cf5-8cbd26e0c9fd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:08:21.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37d831d7-674c-4fc3-9cf5-8cbd26e0c9fd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@37cfcce7 2021-07-13T00:08:21.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37d831d7-674c-4fc3-9cf5-8cbd26e0c9fd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:08:21.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37d831d7-674c-4fc3-9cf5-8cbd26e0c9fd PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:08:21.221Z [qtp817348612-18] INFO o.o.s.b.c.i.ComponentBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:08:14.274Z InvocationID=9703b080-fdf2-4dab-b1fc-ebaefb0287ea PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp LifecycleState cannot be defined by user. This field will be overridden by the application 2021-07-13T00:08:21.221Z [qtp817348612-18] INFO o.o.s.b.c.i.ComponentBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:08:14.274Z InvocationID=9703b080-fdf2-4dab-b1fc-ebaefb0287ea PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Version cannot be defined by user. This field will be overridden by the application 2021-07-13T00:08:21.221Z [qtp817348612-18] INFO o.o.s.b.c.i.ComponentBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:08:14.274Z InvocationID=9703b080-fdf2-4dab-b1fc-ebaefb0287ea PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Creator cannot be defined by user. This field will be overridden by the application 2021-07-13T00:08:21.221Z [qtp817348612-18] INFO o.o.s.b.c.i.ComponentBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:08:14.274Z InvocationID=9703b080-fdf2-4dab-b1fc-ebaefb0287ea PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Is Highest Version cannot be defined by user. This field will be overridden by the application 2021-07-13T00:08:22.077Z [qtp817348612-18] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Create Resource - validateCapabilityTypesCreate ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:08:14.274Z InvocationID=9703b080-fdf2-4dab-b1fc-ebaefb0287ea ErrorCode=400 Capability Type tosca.capabilities.nfv.ContainerGroupable required is missing 2021-07-13T00:08:22.077Z [qtp817348612-18] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Create Resource - validateCapabilityTypesCreate ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:08:14.274Z InvocationID=9703b080-fdf2-4dab-b1fc-ebaefb0287ea ErrorCode=500 Operation towards database failed. 2021-07-13T00:08:22.161Z [qtp817348612-18] INFO o.o.s.b.m.o.impl.GraphLockOperation AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "osContainerGroup" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "" INVARIANT_UUID = "" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "" CURR_STATE = "CERTIFIED" STATUS = "400" DESC = "SVC4115: Error: Invalid Content. Missing Capability Type 'tosca.capabilities.nfv.ContainerGroupable'." TOSCA_NODE_TYPE = "tosca.nodes.nfv.Vdu.osContainerGroup" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Create Resource - validateCapabilityTypesCreate ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:08:14.274Z InvocationID=9703b080-fdf2-4dab-b1fc-ebaefb0287ea ErrorCode=500 lock resource with name Oscontainergroup 2021-07-13T00:08:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:08:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:08:23.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0914464-aa70-4292-b2f7-77ba348ddc34 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:08:23.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0914464-aa70-4292-b2f7-77ba348ddc34 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@37b9d994 2021-07-13T00:08:23.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0914464-aa70-4292-b2f7-77ba348ddc34 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:08:23.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0914464-aa70-4292-b2f7-77ba348ddc34 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:08:24.600Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:08:26.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c45dea41-70d6-46d6-ac3e-8d8704e941d8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:08:26.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c45dea41-70d6-46d6-ac3e-8d8704e941d8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2dcdbcc7 2021-07-13T00:08:26.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c45dea41-70d6-46d6-ac3e-8d8704e941d8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:08:26.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c45dea41-70d6-46d6-ac3e-8d8704e941d8 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:08:27.312Z [qtp817348612-18] WARN o.j.d.l.c.ConsistentKeyLocker AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "osContainerGroup" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "e6e09ccf-67b4-4d82-99cc-00ea8b7472b0" INVARIANT_UUID = "6b294bb0-d31d-438c-b18f-05d52a80aa59" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "" CURR_STATE = "CERTIFIED" STATUS = "201" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.Vdu.osContainerGroup" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Create Resource - validateCapabilityTypesCreate ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=e6e09ccf-67b4-4d82-99cc-00ea8b7472b0 TargetServiceName=null EntryTimestamp=2021-07-13T00:08:14.274Z InvocationID=9703b080-fdf2-4dab-b1fc-ebaefb0287ea ErrorCode=500 Lock write succeeded but took too long: duration PT1.047168S exceeded limit PT0.5S 2021-07-13T00:08:30.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3101234b-22c8-4c4b-b751-7ade1f76954a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:08:30.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3101234b-22c8-4c4b-b751-7ade1f76954a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4f98d3a7 2021-07-13T00:08:30.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3101234b-22c8-4c4b-b751-7ade1f76954a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:08:30.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3101234b-22c8-4c4b-b751-7ade1f76954a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:08:30.921Z [qtp817348612-18] INFO o.j.d.util.BackendOperation AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "osContainerGroup" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "e6e09ccf-67b4-4d82-99cc-00ea8b7472b0" INVARIANT_UUID = "6b294bb0-d31d-438c-b18f-05d52a80aa59" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "" CURR_STATE = "CERTIFIED" STATUS = "201" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.Vdu.osContainerGroup" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Create Resource - validateCapabilityTypesCreate ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=e6e09ccf-67b4-4d82-99cc-00ea8b7472b0 TargetServiceName=null EntryTimestamp=2021-07-13T00:08:14.274Z InvocationID=9703b080-fdf2-4dab-b1fc-ebaefb0287ea ErrorCode=500 Temporary exception during backend operation [CacheMutation]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateManyUnlogged(CQLStoreManager.java:504) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateMany(CQLStoreManager.java:435) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStoreManager.mutateMany(ExpectedValueCheckingStoreManager.java:76) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:94) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:91) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:91) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:139) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:196) at org.janusgraph.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:134) at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:733) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1379) at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.doCommit(JanusGraphBlueprintsGraph.java:298) at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:104) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.commit(JanusGraphClient.java:250) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.unlockNode(JanusGraphGenericDao.java:1272) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.releaseElement(JanusGraphGenericDao.java:1284) at org.openecomp.sdc.be.model.operations.impl.GraphLockOperation.unlockComponentByName(GraphLockOperation.java:71) at org.openecomp.sdc.be.components.impl.ResourceBusinessLogic.createResourceByDao(ResourceBusinessLogic.java:3765) at org.openecomp.sdc.be.components.impl.ResourceBusinessLogic.createResourceByImport(ResourceBusinessLogic.java:3364) at org.openecomp.sdc.be.components.impl.ResourceBusinessLogic.createOrUpdateResourceByImport(ResourceBusinessLogic.java:3319) at org.openecomp.sdc.be.components.impl.ResourceImportManager.importCertifiedResource(ResourceImportManager.java:188) at org.openecomp.sdc.be.components.impl.ResourceImportManager.importNormativeResource(ResourceImportManager.java:143) at org.openecomp.sdc.be.servlets.AbstractValidationsServlet.handleImport(AbstractValidationsServlet.java:659) at org.openecomp.sdc.be.servlets.ResourceUploadServlet.uploadMultipart_aroundBody0(ResourceUploadServlet.java:131) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$AjcClosure1.run(ResourceUploadServlet.java:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:212) at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) at com.jcabi.aspects.aj.MethodLogger.wrapClass(MethodLogger.java:136) at org.openecomp.sdc.be.servlets.ResourceUploadServlet.uploadMultipart(ResourceUploadServlet.java:109) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$$FastClassBySpringCGLIB$$6732af97.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$$EnhancerBySpringCGLIB$$66c7ac05.uploadMultipart() at jdk.internal.reflect.GeneratedMethodAccessor368.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1411) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:226) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.togglz.servlet.TogglzFilter.doFilter(TogglzFilter.java:100) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.ReqValidationFilter.doFilter(ReqValidationFilter.java:72) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.GatewayFilter.doFilter(GatewayFilter.java:71) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1630) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:501) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:113) at com.datastax.driver.core.Responses$Error.asException(Responses.java:167) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:87) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:08:31.916Z [qtp817348612-18] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "osContainerGroup" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "e6e09ccf-67b4-4d82-99cc-00ea8b7472b0" INVARIANT_UUID = "6b294bb0-d31d-438c-b18f-05d52a80aa59" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "" CURR_STATE = "CERTIFIED" STATUS = "201" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.Vdu.osContainerGroup" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=Create Resource - validateCapabilityTypesCreate ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=e6e09ccf-67b4-4d82-99cc-00ea8b7472b0 TargetServiceName=null EntryTimestamp=2021-07-13T00:08:14.274Z InvocationID=9703b080-fdf2-4dab-b1fc-ebaefb0287ea ErrorCode=500 The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:08:31.918Z [qtp817348612-18] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "osContainerGroup" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "e6e09ccf-67b4-4d82-99cc-00ea8b7472b0" INVARIANT_UUID = "6b294bb0-d31d-438c-b18f-05d52a80aa59" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "" CURR_STATE = "CERTIFIED" STATUS = "201" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.Vdu.osContainerGroup" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=e6e09ccf-67b4-4d82-99cc-00ea8b7472b0 TargetServiceName=null EntryTimestamp=2021-07-13T00:08:14.274Z InvocationID=9703b080-fdf2-4dab-b1fc-ebaefb0287ea ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:08:32.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=292e6b41-de09-4971-af32-508c98b8d9cf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:08:32.769Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=292e6b41-de09-4971-af32-508c98b8d9cf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6798647 2021-07-13T00:08:32.769Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=292e6b41-de09-4971-af32-508c98b8d9cf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:08:32.769Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=292e6b41-de09-4971-af32-508c98b8d9cf PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:08:35.776Z [qtp817348612-18] INFO o.o.s.b.m.o.impl.GraphLockOperation AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "osContainerGroup" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "e6e09ccf-67b4-4d82-99cc-00ea8b7472b0" INVARIANT_UUID = "6b294bb0-d31d-438c-b18f-05d52a80aa59" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "" CURR_STATE = "CERTIFIED" STATUS = "201" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.Vdu.osContainerGroup" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=e6e09ccf-67b4-4d82-99cc-00ea8b7472b0 TargetServiceName=null EntryTimestamp=2021-07-13T00:08:14.274Z InvocationID=9703b080-fdf2-4dab-b1fc-ebaefb0287ea ErrorCode=500 lock resource with id eb3bf97b-9887-44ae-881a-fca7074f7bde 2021-07-13T00:08:35.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fceed1eb-5ab1-4691-9685-0dbfd64da384 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:08:35.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fceed1eb-5ab1-4691-9685-0dbfd64da384 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f758a65 2021-07-13T00:08:35.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fceed1eb-5ab1-4691-9685-0dbfd64da384 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:08:35.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fceed1eb-5ab1-4691-9685-0dbfd64da384 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:08:37.918Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:08:37.929Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:08:38.664Z [qtp817348612-18] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "osContainerGroup" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "e6e09ccf-67b4-4d82-99cc-00ea8b7472b0" INVARIANT_UUID = "6b294bb0-d31d-438c-b18f-05d52a80aa59" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "eb3bf97b-9887-44ae-881a-fca7074f7bde.assettoscatemplate" CURR_ARTIFACT_UUID = "eb3bf97b-9887-44ae-881a-fca7074f7bde.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-Oscontainergroup-template.yml,0,eb3bf97b-9887-44ae-881a-fca7074f7bde.assettoscatemplate,0 " STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=90dcfcc1-1f8b-492a-b727-b77e5327c8ea TargetServiceName=null EntryTimestamp=2021-07-13T00:08:14.274Z InvocationID=9703b080-fdf2-4dab-b1fc-ebaefb0287ea ErrorCode=500 Starting to create Csar for component osContainerGroup 2021-07-13T00:08:38.888Z [qtp817348612-18] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "osContainerGroup" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "e6e09ccf-67b4-4d82-99cc-00ea8b7472b0" INVARIANT_UUID = "6b294bb0-d31d-438c-b18f-05d52a80aa59" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "eb3bf97b-9887-44ae-881a-fca7074f7bde.assettoscatemplate" CURR_ARTIFACT_UUID = "eb3bf97b-9887-44ae-881a-fca7074f7bde.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-Oscontainergroup-template.yml,0,eb3bf97b-9887-44ae-881a-fca7074f7bde.assettoscatemplate,0 " STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=90dcfcc1-1f8b-492a-b727-b77e5327c8ea TargetServiceName=null EntryTimestamp=2021-07-13T00:08:14.274Z InvocationID=9703b080-fdf2-4dab-b1fc-ebaefb0287ea ErrorCode=500 Ended create Csar for component osContainerGroup 2021-07-13T00:08:38.889Z [qtp817348612-18] INFO o.o.s.b.m.o.impl.GraphLockOperation AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "osContainerGroup" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "e6e09ccf-67b4-4d82-99cc-00ea8b7472b0" INVARIANT_UUID = "6b294bb0-d31d-438c-b18f-05d52a80aa59" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "eb3bf97b-9887-44ae-881a-fca7074f7bde.assettoscatemplate" CURR_ARTIFACT_UUID = "eb3bf97b-9887-44ae-881a-fca7074f7bde.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-Oscontainergroup-template.yml,0,eb3bf97b-9887-44ae-881a-fca7074f7bde.assettoscatemplate,0 " STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=90dcfcc1-1f8b-492a-b727-b77e5327c8ea TargetServiceName=null EntryTimestamp=2021-07-13T00:08:14.274Z InvocationID=9703b080-fdf2-4dab-b1fc-ebaefb0287ea ErrorCode=500 lock resource with id eb3bf97b-9887-44ae-881a-fca7074f7bde 2021-07-13T00:08:38.969Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1bee811-9a20-4661-b776-905e84f524b9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:08:38.969Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1bee811-9a20-4661-b776-905e84f524b9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3f9e2ac8 2021-07-13T00:08:38.969Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1bee811-9a20-4661-b776-905e84f524b9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:08:38.969Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1bee811-9a20-4661-b776-905e84f524b9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:08:40.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:08:40.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:08:41.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=010899e2-512d-4513-afed-f8ef7788894e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:08:41.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=010899e2-512d-4513-afed-f8ef7788894e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3612d377 2021-07-13T00:08:41.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=010899e2-512d-4513-afed-f8ef7788894e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:08:41.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=010899e2-512d-4513-afed-f8ef7788894e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:08:42.760Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=010899e2-512d-4513-afed-f8ef7788894e ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:08:42.760Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=010899e2-512d-4513-afed-f8ef7788894e ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:08:42.818Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:08:44.924Z [qtp817348612-18] WARN o.j.d.l.c.ConsistentKeyLocker AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "osContainerGroup" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "e6e09ccf-67b4-4d82-99cc-00ea8b7472b0" INVARIANT_UUID = "6b294bb0-d31d-438c-b18f-05d52a80aa59" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "eb3bf97b-9887-44ae-881a-fca7074f7bde.assettoscacsar" CURR_ARTIFACT_UUID = "eb3bf97b-9887-44ae-881a-fca7074f7bde.assettoscacsar" ARTIFACT_DATA = "TOSCA,'assettoscacsar',TOSCA_CSAR,resource-Oscontainergroup-csar.csar,0,eb3bf97b-9887-44ae-881a-fca7074f7bde.assettoscacsar,0 " STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=f5c6b446-4856-4f0b-b3f2-6d4d5b378f74 TargetServiceName=null EntryTimestamp=2021-07-13T00:08:14.274Z InvocationID=9703b080-fdf2-4dab-b1fc-ebaefb0287ea ErrorCode=500 Temporary exception during lock write org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateManyUnlogged(CQLStoreManager.java:504) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateMany(CQLStoreManager.java:435) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.mutate(CQLKeyColumnValueStore.java:363) at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.tryWriteLockOnce(ConsistentKeyLocker.java:387) at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.writeSingleLock(ConsistentKeyLocker.java:324) at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.writeSingleLock(ConsistentKeyLocker.java:118) at org.janusgraph.diskstorage.locking.AbstractLocker.writeLock(AbstractLocker.java:304) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStore.acquireLock(ExpectedValueCheckingStore.java:103) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.acquireLock(KCVSProxy.java:52) at org.janusgraph.diskstorage.BackendTransaction.acquireIndexLock(BackendTransaction.java:261) at org.janusgraph.graphdb.database.StandardJanusGraph.prepareCommit(StandardJanusGraph.java:570) at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:717) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1379) at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.doCommit(JanusGraphBlueprintsGraph.java:298) at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:104) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.commit(JanusGraphClient.java:250) at org.openecomp.sdc.be.dao.jsongraph.JanusGraphDao.commit(JanusGraphDao.java:76) at org.openecomp.sdc.be.components.impl.BaseBusinessLogic.unlockComponent(BaseBusinessLogic.java:286) at org.openecomp.sdc.be.components.impl.ArtifactsBusinessLogic.lockComponentAndUpdateArtifact(ArtifactsBusinessLogic.java:877) at org.openecomp.sdc.be.components.impl.ArtifactsBusinessLogic.lambda$generateAndSaveToscaArtifact$3(ArtifactsBusinessLogic.java:355) at fj.data.Either$LeftProjection.bind(Either.java:287) at org.openecomp.sdc.be.components.impl.ArtifactsBusinessLogic.generateAndSaveToscaArtifact(ArtifactsBusinessLogic.java:350) at org.openecomp.sdc.be.components.impl.ComponentBusinessLogic.saveToscaArtifactPayload(ComponentBusinessLogic.java:515) at org.openecomp.sdc.be.components.impl.ComponentBusinessLogic.lambda$populateToscaArtifacts$4(ComponentBusinessLogic.java:500) at org.openecomp.sdc.be.components.impl.ComponentBusinessLogic.saveToscaArtifactAndPopulateToscaArtifactsWithResult(ComponentBusinessLogic.java:145) at org.openecomp.sdc.be.components.impl.ComponentBusinessLogic.populateToscaArtifacts(ComponentBusinessLogic.java:499) at org.openecomp.sdc.be.components.impl.ComponentBusinessLogic.populateToscaArtifacts(ComponentBusinessLogic.java:480) at org.openecomp.sdc.be.components.impl.ResourceBusinessLogic.propagateStateToCertified(ResourceBusinessLogic.java:3258) at org.openecomp.sdc.be.components.impl.ResourceImportManager.importCertifiedResource(ResourceImportManager.java:200) at org.openecomp.sdc.be.components.impl.ResourceImportManager.importNormativeResource(ResourceImportManager.java:143) at org.openecomp.sdc.be.servlets.AbstractValidationsServlet.handleImport(AbstractValidationsServlet.java:659) at org.openecomp.sdc.be.servlets.ResourceUploadServlet.uploadMultipart_aroundBody0(ResourceUploadServlet.java:131) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$AjcClosure1.run(ResourceUploadServlet.java:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:212) at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) at com.jcabi.aspects.aj.MethodLogger.wrapClass(MethodLogger.java:136) at org.openecomp.sdc.be.servlets.ResourceUploadServlet.uploadMultipart(ResourceUploadServlet.java:109) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$$FastClassBySpringCGLIB$$6732af97.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$$EnhancerBySpringCGLIB$$66c7ac05.uploadMultipart() at jdk.internal.reflect.GeneratedMethodAccessor368.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1411) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:226) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.togglz.servlet.TogglzFilter.doFilter(TogglzFilter.java:100) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.ReqValidationFilter.doFilter(ReqValidationFilter.java:72) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.GatewayFilter.doFilter(GatewayFilter.java:71) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1630) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:501) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:113) at com.datastax.driver.core.Responses$Error.asException(Responses.java:167) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:87) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:08:45.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adda047e-3e5c-4e43-ac56-734687c3203d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:08:45.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adda047e-3e5c-4e43-ac56-734687c3203d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7c93204c 2021-07-13T00:08:45.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adda047e-3e5c-4e43-ac56-734687c3203d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:08:45.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adda047e-3e5c-4e43-ac56-734687c3203d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:08:45.862Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=adda047e-3e5c-4e43-ac56-734687c3203d ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:08:45.863Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=adda047e-3e5c-4e43-ac56-734687c3203d ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:08:46.783Z [qtp817348612-18] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "osContainerGroup" RESOURCE_TYPE = "VFC" SERVICE_INSTANCE_ID = "e6e09ccf-67b4-4d82-99cc-00ea8b7472b0" INVARIANT_UUID = "6b294bb0-d31d-438c-b18f-05d52a80aa59" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "eb3bf97b-9887-44ae-881a-fca7074f7bde.assettoscacsar" CURR_ARTIFACT_UUID = "eb3bf97b-9887-44ae-881a-fca7074f7bde.assettoscacsar" ARTIFACT_DATA = "TOSCA,'assettoscacsar',TOSCA_CSAR,resource-Oscontainergroup-csar.csar,0,eb3bf97b-9887-44ae-881a-fca7074f7bde.assettoscacsar,0 " STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR ResponseCode=201 RequestID=null ElapsedTime=32509 PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp StatusCode=COMPLETE TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=f5c6b446-4856-4f0b-b3f2-6d4d5b378f74 TargetServiceName=null EntryTimestamp=2021-07-13T00:08:14.274Z InvocationID=9703b080-fdf2-4dab-b1fc-ebaefb0287ea ErrorCode=500 LogTimestamp=2021-07-13T00:08:46.783409Z ResponseDesc=Created Exiting. 2021-07-13T00:08:46.784Z [qtp817348612-18] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 SC="201" 2021-07-13T00:08:46.872Z [qtp817348612-96] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=8e8b59e1-65ce-4b28-a02c-4595f3bdf744 InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS RemoteHost=10.242.77.79 ServiceName=v1/catalog/upload/multipart ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:08:46.871918Z InvocationID=b34cda3d-48f2-4801-a90d-1c9c6846814d LogTimestamp=2021-07-13T00:08:46.871947Z Entering 2021-07-13T00:08:46.872Z [qtp817348612-96] INFO o.o.sdc.be.filters.BeServletFilter RequestID=8e8b59e1-65ce-4b28-a02c-4595f3bdf744 InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS userId=jh0003 RemoteHost=10.242.77.79 ServiceName=v1/catalog/upload/multipart ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:08:46.871918Z InvocationID=b34cda3d-48f2-4801-a90d-1c9c6846814d LogTimestamp=2021-07-13T00:08:46.871947Z localAddr=10.242.98.212 remoteAddr=10.242.77.79 POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 2021-07-13T00:08:47.670Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1cdf4cc6-3678-483c-baa1-b26ea03e090b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:08:47.670Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1cdf4cc6-3678-483c-baa1-b26ea03e090b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2f66bb19 2021-07-13T00:08:47.670Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1cdf4cc6-3678-483c-baa1-b26ea03e090b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:08:47.670Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1cdf4cc6-3678-483c-baa1-b26ea03e090b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:08:50.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eacbe068-2301-4a6d-8398-2ef0d4676c59 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:08:50.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eacbe068-2301-4a6d-8398-2ef0d4676c59 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@61e01c57 2021-07-13T00:08:50.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eacbe068-2301-4a6d-8398-2ef0d4676c59 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:08:50.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eacbe068-2301-4a6d-8398-2ef0d4676c59 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:08:51.816Z [qtp817348612-96] INFO o.o.s.b.m.o.impl.GraphLockOperation RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:08:46.872Z InvocationID=8b58a4ff-b75a-45e5-9e5f-bc5c769e62d5 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp lock resource with id 9ee472cb-f6fb-4268-8bdc-a834d2b326e6 2021-07-13T00:08:52.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:08:52.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:08:53.562Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32a98050-1ee0-4377-aa00-e7e7e5de277f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:08:53.562Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32a98050-1ee0-4377-aa00-e7e7e5de277f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@41c9f277 2021-07-13T00:08:53.562Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32a98050-1ee0-4377-aa00-e7e7e5de277f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:08:53.562Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32a98050-1ee0-4377-aa00-e7e7e5de277f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:08:54.765Z [qtp817348612-96] INFO o.o.s.b.c.l.LifecycleBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:08:46.872Z InvocationID=8b58a4ff-b75a-45e5-9e5f-bc5c769e62d5 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp owner of resource 9ee472cb-f6fb-4268-8bdc-a834d2b326e6 is jh0003 2021-07-13T00:08:55.460Z [qtp817348612-96] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "6b1c8132-1507-461b-a38c-2f767ef21565" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "2.0" CURR_VERSION = "2.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:08:46.872Z InvocationID=8b58a4ff-b75a-45e5-9e5f-bc5c769e62d5 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:08:55.461Z [qtp817348612-96] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "6b1c8132-1507-461b-a38c-2f767ef21565" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "2.0" CURR_VERSION = "2.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true TargetServiceName=null EntryTimestamp=2021-07-13T00:08:46.872Z InvocationID=8b58a4ff-b75a-45e5-9e5f-bc5c769e62d5 ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:08:56.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c6a3d2d-78e2-4fd6-ab1f-1c2f1d673b57 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:08:56.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c6a3d2d-78e2-4fd6-ab1f-1c2f1d673b57 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@243d3b42 2021-07-13T00:08:56.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c6a3d2d-78e2-4fd6-ab1f-1c2f1d673b57 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:08:56.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c6a3d2d-78e2-4fd6-ab1f-1c2f1d673b57 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:08:59.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=515e999d-a16a-47f3-8654-8c791ac138d9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:08:59.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=515e999d-a16a-47f3-8654-8c791ac138d9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3e99e2af 2021-07-13T00:08:59.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=515e999d-a16a-47f3-8654-8c791ac138d9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:08:59.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=515e999d-a16a-47f3-8654-8c791ac138d9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:09:03.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7aad9b95-1b7c-4fe9-93fb-84b4d4e3c2ff PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:09:03.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7aad9b95-1b7c-4fe9-93fb-84b4d4e3c2ff PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5ffa538f 2021-07-13T00:09:03.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7aad9b95-1b7c-4fe9-93fb-84b4d4e3c2ff PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:09:03.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7aad9b95-1b7c-4fe9-93fb-84b4d4e3c2ff PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:09:06.171Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b1c72df-240e-49b9-9387-3c752c3461b8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:09:06.171Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b1c72df-240e-49b9-9387-3c752c3461b8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@512e584d 2021-07-13T00:09:06.171Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b1c72df-240e-49b9-9387-3c752c3461b8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:09:06.171Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b1c72df-240e-49b9-9387-3c752c3461b8 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:09:06.665Z [qtp817348612-96] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "6b1c8132-1507-461b-a38c-2f767ef21565" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "2.0" CURR_VERSION = "2.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=fe6788fb-031f-4130-8053-e4104b13ac33 TargetServiceName=null EntryTimestamp=2021-07-13T00:08:46.872Z InvocationID=8b58a4ff-b75a-45e5-9e5f-bc5c769e62d5 ErrorCode=500 The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:09:06.665Z [qtp817348612-96] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "Checkout" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "6b1c8132-1507-461b-a38c-2f767ef21565" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "2.0" CURR_VERSION = "2.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "CERTIFIED" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" COMMENT = "update by import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=fe6788fb-031f-4130-8053-e4104b13ac33 TargetServiceName=null EntryTimestamp=2021-07-13T00:08:46.872Z InvocationID=8b58a4ff-b75a-45e5-9e5f-bc5c769e62d5 ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:09:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:09:07.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:09:09.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d22bbeca-76f8-4f29-8d6a-13024b55e1a6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:09:09.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d22bbeca-76f8-4f29-8d6a-13024b55e1a6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c849e01 2021-07-13T00:09:09.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d22bbeca-76f8-4f29-8d6a-13024b55e1a6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:09:09.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d22bbeca-76f8-4f29-8d6a-13024b55e1a6 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:09:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:09:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:09:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:09:10.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:09:10.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:09:12.069Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=d22bbeca-76f8-4f29-8d6a-13024b55e1a6 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:09:12.070Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=d22bbeca-76f8-4f29-8d6a-13024b55e1a6 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:09:12.070Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=d22bbeca-76f8-4f29-8d6a-13024b55e1a6 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:09:12.070Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=d22bbeca-76f8-4f29-8d6a-13024b55e1a6 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:09:13.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37caa6d9-1d66-4846-83da-3f8e4ff0f3e8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:09:13.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37caa6d9-1d66-4846-83da-3f8e4ff0f3e8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6d798ba4 2021-07-13T00:09:13.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37caa6d9-1d66-4846-83da-3f8e4ff0f3e8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:09:13.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37caa6d9-1d66-4846-83da-3f8e4ff0f3e8 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:09:13.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:09:14.821Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=37caa6d9-1d66-4846-83da-3f8e4ff0f3e8 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:09:14.821Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=37caa6d9-1d66-4846-83da-3f8e4ff0f3e8 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:09:15.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bd38151a-615f-4cb2-a2a0-80afe5cd803b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:09:15.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bd38151a-615f-4cb2-a2a0-80afe5cd803b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@43dd1bd0 2021-07-13T00:09:15.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bd38151a-615f-4cb2-a2a0-80afe5cd803b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:09:15.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bd38151a-615f-4cb2-a2a0-80afe5cd803b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:09:15.862Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:09:18.867Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=bd38151a-615f-4cb2-a2a0-80afe5cd803b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:09:18.868Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=bd38151a-615f-4cb2-a2a0-80afe5cd803b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:09:18.868Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=bd38151a-615f-4cb2-a2a0-80afe5cd803b ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:09:18.868Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=bd38151a-615f-4cb2-a2a0-80afe5cd803b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:09:20.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ee9e7a6-a1f6-4973-8588-f52457a11dc1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:09:20.369Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ee9e7a6-a1f6-4973-8588-f52457a11dc1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7b91f13c 2021-07-13T00:09:20.369Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ee9e7a6-a1f6-4973-8588-f52457a11dc1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:09:20.369Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ee9e7a6-a1f6-4973-8588-f52457a11dc1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:09:20.863Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=7ee9e7a6-a1f6-4973-8588-f52457a11dc1 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:09:20.863Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=7ee9e7a6-a1f6-4973-8588-f52457a11dc1 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:09:22.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a1d65df-49c7-422e-a9af-4b226e5c783c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:09:22.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a1d65df-49c7-422e-a9af-4b226e5c783c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2cad961f 2021-07-13T00:09:22.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a1d65df-49c7-422e-a9af-4b226e5c783c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:09:22.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a1d65df-49c7-422e-a9af-4b226e5c783c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:09:22.875Z [qtp817348612-96] INFO o.o.s.b.m.o.impl.GraphLockOperation AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "6b1c8132-1507-461b-a38c-2f767ef21565" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "2.1" CURR_VERSION = "2.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" STATUS = "200" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.VduCp" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=fe6788fb-031f-4130-8053-e4104b13ac33 TargetServiceName=null EntryTimestamp=2021-07-13T00:08:46.872Z InvocationID=8b58a4ff-b75a-45e5-9e5f-bc5c769e62d5 ErrorCode=500 lock resource with id 5387fb89-68a1-4ba7-b210-3145be9a83e8 2021-07-13T00:09:22.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:09:22.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:09:23.571Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f2b2e50f-4690-438c-a91d-2844999dcff6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:09:23.571Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f2b2e50f-4690-438c-a91d-2844999dcff6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7050bc1 2021-07-13T00:09:23.572Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f2b2e50f-4690-438c-a91d-2844999dcff6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:09:23.572Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f2b2e50f-4690-438c-a91d-2844999dcff6 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:09:25.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:09:25.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:09:26.577Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=f2b2e50f-4690-438c-a91d-2844999dcff6 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:09:26.577Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=f2b2e50f-4690-438c-a91d-2844999dcff6 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:09:26.578Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f2b2e50f-4690-438c-a91d-2844999dcff6 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:09:26.578Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f2b2e50f-4690-438c-a91d-2844999dcff6 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:09:27.681Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:09:27.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4d7701e-e782-4546-94b4-be3eadad0529 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:09:27.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4d7701e-e782-4546-94b4-be3eadad0529 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@73e9120c 2021-07-13T00:09:27.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4d7701e-e782-4546-94b4-be3eadad0529 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:09:27.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4d7701e-e782-4546-94b4-be3eadad0529 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:09:28.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:09:30.901Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=e4d7701e-e782-4546-94b4-be3eadad0529 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:09:30.901Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=e4d7701e-e782-4546-94b4-be3eadad0529 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:09:30.902Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=e4d7701e-e782-4546-94b4-be3eadad0529 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:09:30.902Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=e4d7701e-e782-4546-94b4-be3eadad0529 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:09:31.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:09:32.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4bd546fa-51a0-45dd-902a-5892e46f80be PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:09:32.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4bd546fa-51a0-45dd-902a-5892e46f80be PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4a538979 2021-07-13T00:09:32.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4bd546fa-51a0-45dd-902a-5892e46f80be PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:09:32.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4bd546fa-51a0-45dd-902a-5892e46f80be PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:09:34.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:09:35.096Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=4bd546fa-51a0-45dd-902a-5892e46f80be ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:09:35.096Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=4bd546fa-51a0-45dd-902a-5892e46f80be ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:09:36.072Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=77c8902d-f4c8-4139-949e-691e40cfa5b4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:09:36.072Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=77c8902d-f4c8-4139-949e-691e40cfa5b4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5ee5cf3b 2021-07-13T00:09:36.072Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=77c8902d-f4c8-4139-949e-691e40cfa5b4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:09:36.072Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=77c8902d-f4c8-4139-949e-691e40cfa5b4 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:09:37.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:09:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:09:37.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:09:39.111Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=77c8902d-f4c8-4139-949e-691e40cfa5b4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:09:39.111Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=77c8902d-f4c8-4139-949e-691e40cfa5b4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:09:40.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:09:40.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fbd385b7-6b06-4be8-b2b3-d21e6c77f443 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:09:40.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fbd385b7-6b06-4be8-b2b3-d21e6c77f443 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7582fe4c 2021-07-13T00:09:40.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fbd385b7-6b06-4be8-b2b3-d21e6c77f443 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:09:40.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fbd385b7-6b06-4be8-b2b3-d21e6c77f443 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:09:43.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:09:43.497Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=fbd385b7-6b06-4be8-b2b3-d21e6c77f443 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:09:43.497Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=fbd385b7-6b06-4be8-b2b3-d21e6c77f443 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:09:43.514Z [qtp817348612-96] INFO o.j.d.util.BackendOperation AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "6b1c8132-1507-461b-a38c-2f767ef21565" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "2.1" CURR_VERSION = "2.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" STATUS = "200" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.VduCp" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=fe6788fb-031f-4130-8053-e4104b13ac33 TargetServiceName=null EntryTimestamp=2021-07-13T00:08:46.872Z InvocationID=8b58a4ff-b75a-45e5-9e5f-bc5c769e62d5 ErrorCode=500 Temporary exception during backend operation [CacheMutation]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateManyUnlogged(CQLStoreManager.java:504) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateMany(CQLStoreManager.java:435) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStoreManager.mutateMany(ExpectedValueCheckingStoreManager.java:76) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:94) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:91) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:91) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:139) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:196) at org.janusgraph.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:134) at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:733) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1379) at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.doCommit(JanusGraphBlueprintsGraph.java:298) at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:104) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.commit(JanusGraphClient.java:250) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.lockNode(JanusGraphGenericDao.java:1130) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.lockElement(JanusGraphGenericDao.java:1117) at org.openecomp.sdc.be.model.operations.impl.GraphLockOperation.lockComponent(GraphLockOperation.java:51) at org.openecomp.sdc.be.components.lifecycle.LifecycleBusinessLogic.lockComponent(LifecycleBusinessLogic.java:307) at org.openecomp.sdc.be.components.lifecycle.LifecycleBusinessLogic.changeComponentState(LifecycleBusinessLogic.java:155) at org.openecomp.sdc.be.components.lifecycle.LifecycleBusinessLogic.changeState(LifecycleBusinessLogic.java:128) at org.openecomp.sdc.be.components.impl.ResourceBusinessLogic.nodeFullCertification(ResourceBusinessLogic.java:3281) at org.openecomp.sdc.be.components.impl.ResourceBusinessLogic.propagateStateToCertified(ResourceBusinessLogic.java:3261) at org.openecomp.sdc.be.components.impl.ResourceImportManager.importCertifiedResource(ResourceImportManager.java:200) at org.openecomp.sdc.be.components.impl.ResourceImportManager.importNormativeResource(ResourceImportManager.java:143) at org.openecomp.sdc.be.servlets.AbstractValidationsServlet.handleImport(AbstractValidationsServlet.java:659) at org.openecomp.sdc.be.servlets.ResourceUploadServlet.uploadMultipart_aroundBody0(ResourceUploadServlet.java:131) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$AjcClosure1.run(ResourceUploadServlet.java:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:212) at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) at com.jcabi.aspects.aj.MethodLogger.wrapClass(MethodLogger.java:136) at org.openecomp.sdc.be.servlets.ResourceUploadServlet.uploadMultipart(ResourceUploadServlet.java:109) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$$FastClassBySpringCGLIB$$6732af97.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$$EnhancerBySpringCGLIB$$66c7ac05.uploadMultipart() at jdk.internal.reflect.GeneratedMethodAccessor368.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1411) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:226) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.togglz.servlet.TogglzFilter.doFilter(TogglzFilter.java:100) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.ReqValidationFilter.doFilter(ReqValidationFilter.java:72) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.GatewayFilter.doFilter(GatewayFilter.java:71) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1630) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:501) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:113) at com.datastax.driver.core.Responses$Error.asException(Responses.java:167) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:87) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:09:44.325Z [qtp817348612-96] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "6b1c8132-1507-461b-a38c-2f767ef21565" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "2.1" CURR_VERSION = "2.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" STATUS = "200" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.VduCp" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=fe6788fb-031f-4130-8053-e4104b13ac33 TargetServiceName=null EntryTimestamp=2021-07-13T00:08:46.872Z InvocationID=8b58a4ff-b75a-45e5-9e5f-bc5c769e62d5 ErrorCode=500 owner of resource 5387fb89-68a1-4ba7-b210-3145be9a83e8 is jh0003 2021-07-13T00:09:44.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ec8e017-3d40-4149-b892-d16f48e42728 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:09:44.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ec8e017-3d40-4149-b892-d16f48e42728 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e555eb6 2021-07-13T00:09:44.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ec8e017-3d40-4149-b892-d16f48e42728 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:09:44.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ec8e017-3d40-4149-b892-d16f48e42728 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:09:46.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:09:47.610Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=7ec8e017-3d40-4149-b892-d16f48e42728 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:09:47.611Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=7ec8e017-3d40-4149-b892-d16f48e42728 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:09:48.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7a68a93-0626-45b5-8a6e-a3cc1b0a882b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:09:48.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7a68a93-0626-45b5-8a6e-a3cc1b0a882b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7a5527e0 2021-07-13T00:09:48.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7a68a93-0626-45b5-8a6e-a3cc1b0a882b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:09:48.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c7a68a93-0626-45b5-8a6e-a3cc1b0a882b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:09:49.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:09:51.608Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=c7a68a93-0626-45b5-8a6e-a3cc1b0a882b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:09:51.609Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=c7a68a93-0626-45b5-8a6e-a3cc1b0a882b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:09:52.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:09:52.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6792f82e-c176-46d5-9979-54820210fdc1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:09:52.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6792f82e-c176-46d5-9979-54820210fdc1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6e5a76c6 2021-07-13T00:09:52.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6792f82e-c176-46d5-9979-54820210fdc1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:09:52.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6792f82e-c176-46d5-9979-54820210fdc1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:09:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:09:52.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:09:53.214Z [JanusGraph-Health-Check-Thread] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.execute(VertexCentricQueryBuilder.java:73) at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.properties(VertexCentricQueryBuilder.java:100) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Vertex.property(Vertex.java:70) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:341) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:336) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ... 4 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:09:53.214Z [ApplicationDataTypeCacheThread-1] INFO o.j.d.util.BackendOperation RequestID=null ErrorCategory=INFO ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=500 TargetEntity=ReplaceDataTypesCache Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Element.keys(Element.java:69) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getProperties(JanusGraphGenericDao.java:1491) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getByCriteria(JanusGraphGenericDao.java:759) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAllDataTypeNodes(PropertyOperation.java:1566) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.findAllDataTypesLazy(ApplicationDataTypeCache.java:309) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.hasDataTypesChanged(ApplicationDataTypeCache.java:220) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.run(ApplicationDataTypeCache.java:200) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:308) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:422) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 26 common frames omitted 2021-07-13T00:09:53.215Z [qtp817348612-96] INFO o.j.d.util.BackendOperation AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "6b1c8132-1507-461b-a38c-2f767ef21565" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "2.1" CURR_VERSION = "2.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" STATUS = "200" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.VduCp" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=fe6788fb-031f-4130-8053-e4104b13ac33 TargetServiceName=null EntryTimestamp=2021-07-13T00:08:46.872Z InvocationID=8b58a4ff-b75a-45e5-9e5f-bc5c769e62d5 ErrorCode=500 Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.openecomp.sdc.be.dao.jsongraph.JanusGraphDao.deleteEdge(JanusGraphDao.java:809) at org.openecomp.sdc.be.dao.jsongraph.JanusGraphDao.deleteEdge(JanusGraphDao.java:796) at org.openecomp.sdc.be.model.jsonjanusgraph.operations.ToscaElementLifecycleOperation.changeStateToCheckedIn(ToscaElementLifecycleOperation.java:1011) at org.openecomp.sdc.be.model.jsonjanusgraph.operations.ToscaElementLifecycleOperation.checkinToscaELement(ToscaElementLifecycleOperation.java:976) at org.openecomp.sdc.be.model.jsonjanusgraph.operations.ToscaElementLifecycleOperation.lambda$checkinToscaELement$3(ToscaElementLifecycleOperation.java:125) at fj.data.Either$LeftProjection.bind(Either.java:287) at org.openecomp.sdc.be.model.jsonjanusgraph.operations.ToscaElementLifecycleOperation.checkinToscaELement(ToscaElementLifecycleOperation.java:124) at org.openecomp.sdc.be.components.lifecycle.CheckinTransition.changeState(CheckinTransition.java:82) at org.openecomp.sdc.be.components.lifecycle.LifecycleBusinessLogic.changeState(LifecycleBusinessLogic.java:258) at org.openecomp.sdc.be.components.lifecycle.LifecycleBusinessLogic.checkInBeforeCertifyIfNeeded(LifecycleBusinessLogic.java:224) at org.openecomp.sdc.be.components.lifecycle.LifecycleBusinessLogic.changeComponentState(LifecycleBusinessLogic.java:185) at org.openecomp.sdc.be.components.lifecycle.LifecycleBusinessLogic.changeState(LifecycleBusinessLogic.java:128) at org.openecomp.sdc.be.components.impl.ResourceBusinessLogic.nodeFullCertification(ResourceBusinessLogic.java:3281) at org.openecomp.sdc.be.components.impl.ResourceBusinessLogic.propagateStateToCertified(ResourceBusinessLogic.java:3261) at org.openecomp.sdc.be.components.impl.ResourceImportManager.importCertifiedResource(ResourceImportManager.java:200) at org.openecomp.sdc.be.components.impl.ResourceImportManager.importNormativeResource(ResourceImportManager.java:143) at org.openecomp.sdc.be.servlets.AbstractValidationsServlet.handleImport(AbstractValidationsServlet.java:659) at org.openecomp.sdc.be.servlets.ResourceUploadServlet.uploadMultipart_aroundBody0(ResourceUploadServlet.java:131) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$AjcClosure1.run(ResourceUploadServlet.java:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:212) at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) at com.jcabi.aspects.aj.MethodLogger.wrapClass(MethodLogger.java:136) at org.openecomp.sdc.be.servlets.ResourceUploadServlet.uploadMultipart(ResourceUploadServlet.java:109) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$$FastClassBySpringCGLIB$$6732af97.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$$EnhancerBySpringCGLIB$$66c7ac05.uploadMultipart() at jdk.internal.reflect.GeneratedMethodAccessor368.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1411) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:226) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.togglz.servlet.TogglzFilter.doFilter(TogglzFilter.java:100) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.ReqValidationFilter.doFilter(ReqValidationFilter.java:72) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.GatewayFilter.doFilter(GatewayFilter.java:71) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1630) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:501) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:308) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:422) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 26 common frames omitted 2021-07-13T00:09:54.162Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6792f82e-c176-46d5-9979-54820210fdc1 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:09:54.162Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=6792f82e-c176-46d5-9979-54820210fdc1 ErrorCategory=ERROR ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:09:55.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:09:55.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=152e91e9-10c4-49de-a8ed-b6dd171dcb42 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:09:55.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=152e91e9-10c4-49de-a8ed-b6dd171dcb42 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b53b3f7 2021-07-13T00:09:55.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=152e91e9-10c4-49de-a8ed-b6dd171dcb42 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:09:55.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=152e91e9-10c4-49de-a8ed-b6dd171dcb42 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:09:58.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:09:58.465Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=152e91e9-10c4-49de-a8ed-b6dd171dcb42 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:09:58.466Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=152e91e9-10c4-49de-a8ed-b6dd171dcb42 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:09:58.466Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=152e91e9-10c4-49de-a8ed-b6dd171dcb42 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:09:58.466Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=152e91e9-10c4-49de-a8ed-b6dd171dcb42 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:09:59.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2924a2ab-f103-4169-8a60-8fc40e47e0f5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:09:59.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2924a2ab-f103-4169-8a60-8fc40e47e0f5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@70e82391 2021-07-13T00:09:59.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2924a2ab-f103-4169-8a60-8fc40e47e0f5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:09:59.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2924a2ab-f103-4169-8a60-8fc40e47e0f5 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:10:01.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:10:01.814Z [JanusGraph-Health-Check-Thread] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [VertexIndexQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:399) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:396) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.indexQuery(BackendTransaction.java:396) at org.janusgraph.graphdb.query.graph.MultiKeySliceQuery.execute(MultiKeySliceQuery.java:51) at org.janusgraph.graphdb.database.IndexSerializer.query(IndexSerializer.java:512) at org.janusgraph.graphdb.util.SubqueryIterator.(SubqueryIterator.java:66) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1295) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1190) at org.janusgraph.graphdb.query.QueryProcessor$LimitAdjustingIterator.getNewIterator(QueryProcessor.java:194) at org.janusgraph.graphdb.query.LimitAdjustingIterator.hasNext(LimitAdjustingIterator.java:68) at org.janusgraph.graphdb.query.ResultSetIterator.nextInternal(ResultSetIterator.java:54) at org.janusgraph.graphdb.query.ResultSetIterator.(ResultSetIterator.java:44) at org.janusgraph.graphdb.query.QueryProcessor.iterator(QueryProcessor.java:66) at com.google.common.collect.Iterables$4.iterator(Iterables.java:559) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:340) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:336) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ... 4 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:308) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:422) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 26 common frames omitted 2021-07-13T00:10:01.815Z [qtp817348612-96] INFO o.j.d.util.BackendOperation AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "6b1c8132-1507-461b-a38c-2f767ef21565" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "2.1" CURR_VERSION = "2.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKOUT" STATUS = "200" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.VduCp" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=fe6788fb-031f-4130-8053-e4104b13ac33 TargetServiceName=null EntryTimestamp=2021-07-13T00:08:46.872Z InvocationID=8b58a4ff-b75a-45e5-9e5f-bc5c769e62d5 ErrorCode=500 Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.execute(VertexCentricQueryBuilder.java:73) at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.properties(VertexCentricQueryBuilder.java:100) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Vertex.property(Vertex.java:70) at org.apache.tinkerpop.gremlin.structure.Vertex.property(Vertex.java:38) at org.apache.tinkerpop.gremlin.structure.Element.value(Element.java:94) at org.openecomp.sdc.be.dao.jsongraph.JanusGraphDao.deleteEdge(JanusGraphDao.java:815) at org.openecomp.sdc.be.dao.jsongraph.JanusGraphDao.deleteEdge(JanusGraphDao.java:796) at org.openecomp.sdc.be.model.jsonjanusgraph.operations.ToscaElementLifecycleOperation.changeStateToCheckedIn(ToscaElementLifecycleOperation.java:1011) at org.openecomp.sdc.be.model.jsonjanusgraph.operations.ToscaElementLifecycleOperation.checkinToscaELement(ToscaElementLifecycleOperation.java:976) at org.openecomp.sdc.be.model.jsonjanusgraph.operations.ToscaElementLifecycleOperation.lambda$checkinToscaELement$3(ToscaElementLifecycleOperation.java:125) at fj.data.Either$LeftProjection.bind(Either.java:287) at org.openecomp.sdc.be.model.jsonjanusgraph.operations.ToscaElementLifecycleOperation.checkinToscaELement(ToscaElementLifecycleOperation.java:124) at org.openecomp.sdc.be.components.lifecycle.CheckinTransition.changeState(CheckinTransition.java:82) at org.openecomp.sdc.be.components.lifecycle.LifecycleBusinessLogic.changeState(LifecycleBusinessLogic.java:258) at org.openecomp.sdc.be.components.lifecycle.LifecycleBusinessLogic.checkInBeforeCertifyIfNeeded(LifecycleBusinessLogic.java:224) at org.openecomp.sdc.be.components.lifecycle.LifecycleBusinessLogic.changeComponentState(LifecycleBusinessLogic.java:185) at org.openecomp.sdc.be.components.lifecycle.LifecycleBusinessLogic.changeState(LifecycleBusinessLogic.java:128) at org.openecomp.sdc.be.components.impl.ResourceBusinessLogic.nodeFullCertification(ResourceBusinessLogic.java:3281) at org.openecomp.sdc.be.components.impl.ResourceBusinessLogic.propagateStateToCertified(ResourceBusinessLogic.java:3261) at org.openecomp.sdc.be.components.impl.ResourceImportManager.importCertifiedResource(ResourceImportManager.java:200) at org.openecomp.sdc.be.components.impl.ResourceImportManager.importNormativeResource(ResourceImportManager.java:143) at org.openecomp.sdc.be.servlets.AbstractValidationsServlet.handleImport(AbstractValidationsServlet.java:659) at org.openecomp.sdc.be.servlets.ResourceUploadServlet.uploadMultipart_aroundBody0(ResourceUploadServlet.java:131) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$AjcClosure1.run(ResourceUploadServlet.java:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:212) at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) at com.jcabi.aspects.aj.MethodLogger.wrapClass(MethodLogger.java:136) at org.openecomp.sdc.be.servlets.ResourceUploadServlet.uploadMultipart(ResourceUploadServlet.java:109) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$$FastClassBySpringCGLIB$$6732af97.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$$EnhancerBySpringCGLIB$$66c7ac05.uploadMultipart() at jdk.internal.reflect.GeneratedMethodAccessor368.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1411) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:226) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.togglz.servlet.TogglzFilter.doFilter(TogglzFilter.java:100) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.ReqValidationFilter.doFilter(ReqValidationFilter.java:72) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.GatewayFilter.doFilter(GatewayFilter.java:71) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1630) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:501) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:10:02.425Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=2924a2ab-f103-4169-8a60-8fc40e47e0f5 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:10:02.425Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=2924a2ab-f103-4169-8a60-8fc40e47e0f5 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:10:03.464Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce05d2c8-30b1-4c16-ab45-af54e46c4796 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:10:03.464Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce05d2c8-30b1-4c16-ab45-af54e46c4796 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5275c289 2021-07-13T00:10:03.464Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce05d2c8-30b1-4c16-ab45-af54e46c4796 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:10:03.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ce05d2c8-30b1-4c16-ab45-af54e46c4796 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:10:03.869Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:10:04.465Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=ce05d2c8-30b1-4c16-ab45-af54e46c4796 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:10:04.465Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=ce05d2c8-30b1-4c16-ab45-af54e46c4796 ErrorCategory=ERROR ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:10:05.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2996c89b-6cbf-46ff-bac4-d016f1b77d56 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:10:05.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2996c89b-6cbf-46ff-bac4-d016f1b77d56 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@657a5a96 2021-07-13T00:10:05.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2996c89b-6cbf-46ff-bac4-d016f1b77d56 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:10:05.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2996c89b-6cbf-46ff-bac4-d016f1b77d56 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:10:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:10:07.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:10:08.764Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=2996c89b-6cbf-46ff-bac4-d016f1b77d56 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:10:08.765Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=2996c89b-6cbf-46ff-bac4-d016f1b77d56 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:10:08.765Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=2996c89b-6cbf-46ff-bac4-d016f1b77d56 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:10:08.765Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=2996c89b-6cbf-46ff-bac4-d016f1b77d56 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:10:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:10:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:10:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:10:09.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7be828b-5c16-4c35-8877-215d3e6aee55 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:10:09.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7be828b-5c16-4c35-8877-215d3e6aee55 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64fa30d2 2021-07-13T00:10:09.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7be828b-5c16-4c35-8877-215d3e6aee55 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:10:09.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7be828b-5c16-4c35-8877-215d3e6aee55 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:10:10.618Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f7be828b-5c16-4c35-8877-215d3e6aee55 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:10:10.618Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f7be828b-5c16-4c35-8877-215d3e6aee55 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:10:11.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=222ec7dd-2c93-4e73-b3af-fdbd42281ea0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:10:11.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=222ec7dd-2c93-4e73-b3af-fdbd42281ea0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@340dd5ae 2021-07-13T00:10:11.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=222ec7dd-2c93-4e73-b3af-fdbd42281ea0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:10:11.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=222ec7dd-2c93-4e73-b3af-fdbd42281ea0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:10:13.166Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98bda89d-e417-43f9-89b7-d5bbf0641f8d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:10:13.166Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98bda89d-e417-43f9-89b7-d5bbf0641f8d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3149753 2021-07-13T00:10:13.166Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98bda89d-e417-43f9-89b7-d5bbf0641f8d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:10:13.166Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98bda89d-e417-43f9-89b7-d5bbf0641f8d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:10:14.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b1910b8-113e-4643-8dd6-d36138198bf5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:10:14.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b1910b8-113e-4643-8dd6-d36138198bf5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@36dd5b43 2021-07-13T00:10:14.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b1910b8-113e-4643-8dd6-d36138198bf5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:10:14.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b1910b8-113e-4643-8dd6-d36138198bf5 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:10:16.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:10:16.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:10:17.769Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=5b1910b8-113e-4643-8dd6-d36138198bf5 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:10:17.769Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=5b1910b8-113e-4643-8dd6-d36138198bf5 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:10:17.769Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=5b1910b8-113e-4643-8dd6-d36138198bf5 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:10:17.769Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=5b1910b8-113e-4643-8dd6-d36138198bf5 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:10:18.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0189b390-4d9a-4d20-b469-8f234c41f314 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:10:18.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0189b390-4d9a-4d20-b469-8f234c41f314 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@10a62771 2021-07-13T00:10:18.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0189b390-4d9a-4d20-b469-8f234c41f314 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:10:18.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0189b390-4d9a-4d20-b469-8f234c41f314 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:10:19.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:10:22.012Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=0189b390-4d9a-4d20-b469-8f234c41f314 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:10:22.013Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=0189b390-4d9a-4d20-b469-8f234c41f314 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:10:22.013Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=0189b390-4d9a-4d20-b469-8f234c41f314 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:10:22.013Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=0189b390-4d9a-4d20-b469-8f234c41f314 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:10:22.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:10:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:10:22.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:10:22.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e63a898-4945-4937-8065-6a7af32df568 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:10:22.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e63a898-4945-4937-8065-6a7af32df568 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6dd58879 2021-07-13T00:10:22.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e63a898-4945-4937-8065-6a7af32df568 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:10:22.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e63a898-4945-4937-8065-6a7af32df568 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:10:24.581Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:10:25.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:10:26.013Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=4e63a898-4945-4937-8065-6a7af32df568 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:10:26.013Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=4e63a898-4945-4937-8065-6a7af32df568 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:10:27.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c452705e-2f1f-4c3c-8128-354281d1321d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:10:27.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c452705e-2f1f-4c3c-8128-354281d1321d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1032e53c 2021-07-13T00:10:27.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c452705e-2f1f-4c3c-8128-354281d1321d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:10:27.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c452705e-2f1f-4c3c-8128-354281d1321d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:10:28.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:10:30.212Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=c452705e-2f1f-4c3c-8128-354281d1321d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:10:30.212Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=c452705e-2f1f-4c3c-8128-354281d1321d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:10:31.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:10:32.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0805c242-3afb-4f7d-b4fe-b59823f91a5b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:10:32.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0805c242-3afb-4f7d-b4fe-b59823f91a5b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7a11a955 2021-07-13T00:10:32.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0805c242-3afb-4f7d-b4fe-b59823f91a5b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:10:32.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0805c242-3afb-4f7d-b4fe-b59823f91a5b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:10:34.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:10:35.101Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=0805c242-3afb-4f7d-b4fe-b59823f91a5b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:10:35.102Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=0805c242-3afb-4f7d-b4fe-b59823f91a5b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:10:36.576Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=331906df-2237-4a04-a4c6-93d9329f4b4e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:10:36.576Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=331906df-2237-4a04-a4c6-93d9329f4b4e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3acaf69e 2021-07-13T00:10:36.576Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=331906df-2237-4a04-a4c6-93d9329f4b4e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:10:36.576Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=331906df-2237-4a04-a4c6-93d9329f4b4e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:10:37.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:10:37.917Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:10:37.927Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:10:39.640Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=331906df-2237-4a04-a4c6-93d9329f4b4e ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:10:39.641Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=331906df-2237-4a04-a4c6-93d9329f4b4e ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:10:40.466Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:10:40.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f721d3fa-1451-45b6-bd79-1544b46a2ca3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:10:40.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f721d3fa-1451-45b6-bd79-1544b46a2ca3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@45e8f317 2021-07-13T00:10:40.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f721d3fa-1451-45b6-bd79-1544b46a2ca3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:10:40.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f721d3fa-1451-45b6-bd79-1544b46a2ca3 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:10:43.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:10:43.806Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=f721d3fa-1451-45b6-bd79-1544b46a2ca3 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:10:43.806Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=f721d3fa-1451-45b6-bd79-1544b46a2ca3 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:10:44.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=674565ec-e352-4924-b87f-851d07a160c3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:10:44.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=674565ec-e352-4924-b87f-851d07a160c3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@225fb3c1 2021-07-13T00:10:44.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=674565ec-e352-4924-b87f-851d07a160c3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:10:44.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=674565ec-e352-4924-b87f-851d07a160c3 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:10:46.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:10:47.910Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=674565ec-e352-4924-b87f-851d07a160c3 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:10:47.910Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=674565ec-e352-4924-b87f-851d07a160c3 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:10:49.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d24b7d8c-17aa-4dc8-a06f-1c07476c9f65 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:10:49.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d24b7d8c-17aa-4dc8-a06f-1c07476c9f65 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@37864104 2021-07-13T00:10:49.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d24b7d8c-17aa-4dc8-a06f-1c07476c9f65 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:10:49.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d24b7d8c-17aa-4dc8-a06f-1c07476c9f65 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:10:49.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:10:52.240Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=d24b7d8c-17aa-4dc8-a06f-1c07476c9f65 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:10:52.241Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=d24b7d8c-17aa-4dc8-a06f-1c07476c9f65 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:10:52.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:10:52.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:10:52.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:10:53.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=674ff91e-c484-43d1-bdb2-64865c670a25 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:10:53.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=674ff91e-c484-43d1-bdb2-64865c670a25 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@41e82522 2021-07-13T00:10:53.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=674ff91e-c484-43d1-bdb2-64865c670a25 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:10:53.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=674ff91e-c484-43d1-bdb2-64865c670a25 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:10:54.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac872096-7a3c-4594-aad7-4816fa0fb239 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:10:54.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac872096-7a3c-4594-aad7-4816fa0fb239 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5266072d 2021-07-13T00:10:54.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac872096-7a3c-4594-aad7-4816fa0fb239 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:10:54.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac872096-7a3c-4594-aad7-4816fa0fb239 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:10:55.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:10:55.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aee470ce-2a98-4f77-a622-03901a6f6094 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:10:55.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aee470ce-2a98-4f77-a622-03901a6f6094 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7e58e30b 2021-07-13T00:10:55.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aee470ce-2a98-4f77-a622-03901a6f6094 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:10:55.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aee470ce-2a98-4f77-a622-03901a6f6094 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:10:56.560Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2020aa85-9397-43ad-9fb7-1be784d0401d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:10:56.560Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2020aa85-9397-43ad-9fb7-1be784d0401d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7e1918d7 2021-07-13T00:10:56.560Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2020aa85-9397-43ad-9fb7-1be784d0401d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:10:56.560Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2020aa85-9397-43ad-9fb7-1be784d0401d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:10:57.857Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=43ada304-2c83-4969-be1d-13c7973d27a6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:10:57.857Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=43ada304-2c83-4969-be1d-13c7973d27a6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4659256b 2021-07-13T00:10:57.857Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=43ada304-2c83-4969-be1d-13c7973d27a6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:10:57.857Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=43ada304-2c83-4969-be1d-13c7973d27a6 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:10:58.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:10:58.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3a3c37b-9b96-4a9f-ba62-c42646abee80 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:10:58.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3a3c37b-9b96-4a9f-ba62-c42646abee80 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@24641de2 2021-07-13T00:10:58.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3a3c37b-9b96-4a9f-ba62-c42646abee80 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:10:58.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3a3c37b-9b96-4a9f-ba62-c42646abee80 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:11:00.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f915990-e694-4e23-afc1-2afed18daff2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:11:00.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f915990-e694-4e23-afc1-2afed18daff2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1d9de2de 2021-07-13T00:11:00.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f915990-e694-4e23-afc1-2afed18daff2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:11:00.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f915990-e694-4e23-afc1-2afed18daff2 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:11:01.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=183527be-cbbb-4633-a6d9-c8a90a6cdf5c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:11:01.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=183527be-cbbb-4633-a6d9-c8a90a6cdf5c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2b748e1a 2021-07-13T00:11:01.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=183527be-cbbb-4633-a6d9-c8a90a6cdf5c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:11:01.367Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=183527be-cbbb-4633-a6d9-c8a90a6cdf5c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:11:01.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:11:02.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=126cd4fc-2e5c-4635-b240-4d9c33e0df68 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:11:02.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=126cd4fc-2e5c-4635-b240-4d9c33e0df68 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@24899462 2021-07-13T00:11:02.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=126cd4fc-2e5c-4635-b240-4d9c33e0df68 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:11:02.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=126cd4fc-2e5c-4635-b240-4d9c33e0df68 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:11:03.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=670ed710-f010-4e64-9698-0f5d593e42fc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:11:03.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=670ed710-f010-4e64-9698-0f5d593e42fc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2006161c 2021-07-13T00:11:03.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=670ed710-f010-4e64-9698-0f5d593e42fc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:11:03.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=670ed710-f010-4e64-9698-0f5d593e42fc PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:11:04.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:11:05.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d85f068-85cf-4506-9551-316fec0d96a6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:11:05.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d85f068-85cf-4506-9551-316fec0d96a6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@52ff8bd0 2021-07-13T00:11:05.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d85f068-85cf-4506-9551-316fec0d96a6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:11:05.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d85f068-85cf-4506-9551-316fec0d96a6 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:11:07.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:11:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:11:07.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:11:08.567Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0132cd3-a55d-4147-b40f-6a96dd0bedea PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:11:08.567Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0132cd3-a55d-4147-b40f-6a96dd0bedea PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5df4dbcb 2021-07-13T00:11:08.567Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0132cd3-a55d-4147-b40f-6a96dd0bedea PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:11:08.567Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0132cd3-a55d-4147-b40f-6a96dd0bedea PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:11:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:11:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:11:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:11:10.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:11:11.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bdd7e47d-dc68-4795-af16-e4bedbf64802 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:11:11.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bdd7e47d-dc68-4795-af16-e4bedbf64802 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@35b67e93 2021-07-13T00:11:11.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bdd7e47d-dc68-4795-af16-e4bedbf64802 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:11:11.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bdd7e47d-dc68-4795-af16-e4bedbf64802 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:11:13.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:11:14.471Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d917ff3-abff-4eec-96fc-2889ab44b44d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:11:14.471Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d917ff3-abff-4eec-96fc-2889ab44b44d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@251038cf 2021-07-13T00:11:14.472Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d917ff3-abff-4eec-96fc-2889ab44b44d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:11:14.472Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d917ff3-abff-4eec-96fc-2889ab44b44d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:11:16.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:11:17.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbb019cb-5324-4d32-b0e2-da4140b669ba PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:11:17.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbb019cb-5324-4d32-b0e2-da4140b669ba PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@b1f96b6 2021-07-13T00:11:17.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbb019cb-5324-4d32-b0e2-da4140b669ba PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:11:17.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cbb019cb-5324-4d32-b0e2-da4140b669ba PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:11:19.062Z [qtp817348612-96] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "fe6788fb-031f-4130-8053-e4104b13ac33" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "2.1" CURR_VERSION = "2.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=fe6788fb-031f-4130-8053-e4104b13ac33 TargetServiceName=null EntryTimestamp=2021-07-13T00:08:46.872Z InvocationID=8b58a4ff-b75a-45e5-9e5f-bc5c769e62d5 ErrorCode=500 owner of resource 5387fb89-68a1-4ba7-b210-3145be9a83e8 is jh0003 2021-07-13T00:11:19.062Z [qtp817348612-96] INFO o.o.s.b.c.l.CertificationChangeTransition AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "fe6788fb-031f-4130-8053-e4104b13ac33" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "2.1" CURR_VERSION = "2.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=fe6788fb-031f-4130-8053-e4104b13ac33 TargetServiceName=null EntryTimestamp=2021-07-13T00:08:46.872Z InvocationID=8b58a4ff-b75a-45e5-9e5f-bc5c769e62d5 ErrorCode=500 validate before certification change. resource name=VDU Cp, oldState=NOT_CERTIFIED_CHECKIN, owner userId=jh0003 2021-07-13T00:11:19.062Z [qtp817348612-96] INFO o.o.s.b.c.l.CertificationChangeTransition AuditMessage=ACTION = "Checkin" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "fe6788fb-031f-4130-8053-e4104b13ac33" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "2.1" CURR_VERSION = "2.1" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKOUT" CURR_STATE = "NOT_CERTIFIED_CHECKIN" COMMENT = "certification on import" STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=fe6788fb-031f-4130-8053-e4104b13ac33 TargetServiceName=null EntryTimestamp=2021-07-13T00:08:46.872Z InvocationID=8b58a4ff-b75a-45e5-9e5f-bc5c769e62d5 ErrorCode=500 start performing certification change for resource 5387fb89-68a1-4ba7-b210-3145be9a83e8 2021-07-13T00:11:19.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:11:20.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d260bd4d-4986-456a-a7b1-9f14314b5cd5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:11:20.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d260bd4d-4986-456a-a7b1-9f14314b5cd5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@70ce2e9d 2021-07-13T00:11:20.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d260bd4d-4986-456a-a7b1-9f14314b5cd5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:11:20.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d260bd4d-4986-456a-a7b1-9f14314b5cd5 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:11:21.471Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:11:21.778Z [qtp817348612-96] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "fe6788fb-031f-4130-8053-e4104b13ac33" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "" CURR_VERSION = "2.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "6ad3e8dd-be5e-42f0-8e2d-00c01977d9b7.assettoscatemplate" CURR_ARTIFACT_UUID = "6ad3e8dd-be5e-42f0-8e2d-00c01977d9b7.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VduCp-template.yml,0,6ad3e8dd-be5e-42f0-8e2d-00c01977d9b7.assettoscatemplate,2,6d00cc50-78d5-416a-b4fe-f9f3f316434a" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=f6fd4178-3715-4527-8bf6-638b77d86a7e TargetServiceName=null EntryTimestamp=2021-07-13T00:08:46.872Z InvocationID=8b58a4ff-b75a-45e5-9e5f-bc5c769e62d5 ErrorCode=500 Starting to create Csar for component VDU Cp 2021-07-13T00:11:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:11:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:11:23.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=243663f2-e7c0-4d84-ad1b-3c63cfdba5af PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:11:23.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=243663f2-e7c0-4d84-ad1b-3c63cfdba5af PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@760a84f1 2021-07-13T00:11:23.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=243663f2-e7c0-4d84-ad1b-3c63cfdba5af PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:11:23.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=243663f2-e7c0-4d84-ad1b-3c63cfdba5af PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:11:23.893Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=243663f2-e7c0-4d84-ad1b-3c63cfdba5af ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:11:23.893Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=243663f2-e7c0-4d84-ad1b-3c63cfdba5af ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:11:24.557Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:11:26.373Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6278b27f-0147-48d6-9048-1c2f016c3ab5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:11:26.373Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6278b27f-0147-48d6-9048-1c2f016c3ab5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@52780cbf 2021-07-13T00:11:26.373Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6278b27f-0147-48d6-9048-1c2f016c3ab5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:11:26.373Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6278b27f-0147-48d6-9048-1c2f016c3ab5 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:11:26.376Z [BE-Health-Check-Task] INFO o.a.http.impl.execchain.RetryExec RequestID=6278b27f-0147-48d6-9048-1c2f016c3ab5 PartnerName=SDC.BE I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {s}->https://sdc-onboarding-be.onap:8445: The target server failed to respond 2021-07-13T00:11:26.376Z [BE-Health-Check-Task] INFO o.a.http.impl.execchain.RetryExec RequestID=6278b27f-0147-48d6-9048-1c2f016c3ab5 PartnerName=SDC.BE Retrying request to {s}->https://sdc-onboarding-be.onap:8445 2021-07-13T00:11:26.381Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=6278b27f-0147-48d6-9048-1c2f016c3ab5 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) 2021-07-13T00:11:26.381Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=6278b27f-0147-48d6-9048-1c2f016c3ab5 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:11:27.072Z [qtp817348612-96] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "fe6788fb-031f-4130-8053-e4104b13ac33" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "" CURR_VERSION = "2.1" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "NOT_CERTIFIED_CHECKIN" PREV_ARTIFACT_UUID = "6ad3e8dd-be5e-42f0-8e2d-00c01977d9b7.assettoscatemplate" CURR_ARTIFACT_UUID = "6ad3e8dd-be5e-42f0-8e2d-00c01977d9b7.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VduCp-template.yml,0,6ad3e8dd-be5e-42f0-8e2d-00c01977d9b7.assettoscatemplate,2,6d00cc50-78d5-416a-b4fe-f9f3f316434a" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=f6fd4178-3715-4527-8bf6-638b77d86a7e TargetServiceName=null EntryTimestamp=2021-07-13T00:08:46.872Z InvocationID=8b58a4ff-b75a-45e5-9e5f-bc5c769e62d5 ErrorCode=500 Ended create Csar for component VDU Cp 2021-07-13T00:11:29.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=150080d5-7dbe-4078-91ff-0dcfaa35a1a2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:11:29.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=150080d5-7dbe-4078-91ff-0dcfaa35a1a2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@241ef28f 2021-07-13T00:11:29.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=150080d5-7dbe-4078-91ff-0dcfaa35a1a2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:11:29.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=150080d5-7dbe-4078-91ff-0dcfaa35a1a2 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:11:29.862Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=150080d5-7dbe-4078-91ff-0dcfaa35a1a2 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) 2021-07-13T00:11:29.862Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=150080d5-7dbe-4078-91ff-0dcfaa35a1a2 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:11:32.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bea4ad55-e800-43f3-b492-4718f92b1248 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:11:32.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bea4ad55-e800-43f3-b492-4718f92b1248 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@141e2e68 2021-07-13T00:11:32.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bea4ad55-e800-43f3-b492-4718f92b1248 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:11:32.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bea4ad55-e800-43f3-b492-4718f92b1248 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:11:35.977Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=bea4ad55-e800-43f3-b492-4718f92b1248 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out 2021-07-13T00:11:35.978Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=bea4ad55-e800-43f3-b492-4718f92b1248 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:11:36.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=122e9e5a-0b38-4396-a7db-a976169408ae PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:11:36.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=122e9e5a-0b38-4396-a7db-a976169408ae PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4eb853b8 2021-07-13T00:11:36.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=122e9e5a-0b38-4396-a7db-a976169408ae PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:11:36.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=122e9e5a-0b38-4396-a7db-a976169408ae PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:11:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:11:37.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:11:39.874Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=122e9e5a-0b38-4396-a7db-a976169408ae ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out 2021-07-13T00:11:39.874Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=122e9e5a-0b38-4396-a7db-a976169408ae ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:11:40.157Z [qtp817348612-96] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "fe6788fb-031f-4130-8053-e4104b13ac33" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "2.1" CURR_VERSION = "3.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=f553650e-0093-48d6-9682-6f593964f5de TargetServiceName=null EntryTimestamp=2021-07-13T00:08:46.872Z InvocationID=8b58a4ff-b75a-45e5-9e5f-bc5c769e62d5 ErrorCode=500 The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:11:40.158Z [qtp817348612-96] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "fe6788fb-031f-4130-8053-e4104b13ac33" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "2.1" CURR_VERSION = "3.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=f553650e-0093-48d6-9682-6f593964f5de TargetServiceName=null EntryTimestamp=2021-07-13T00:08:46.872Z InvocationID=8b58a4ff-b75a-45e5-9e5f-bc5c769e62d5 ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:11:40.159Z [qtp817348612-96] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "fe6788fb-031f-4130-8053-e4104b13ac33" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "2.1" CURR_VERSION = "3.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=f553650e-0093-48d6-9682-6f593964f5de TargetServiceName=null EntryTimestamp=2021-07-13T00:08:46.872Z InvocationID=8b58a4ff-b75a-45e5-9e5f-bc5c769e62d5 ErrorCode=500 unlock component 5387fb89-68a1-4ba7-b210-3145be9a83e8 2021-07-13T00:11:40.159Z [qtp817348612-96] INFO o.o.s.b.c.l.LifecycleBusinessLogic AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "fe6788fb-031f-4130-8053-e4104b13ac33" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "2.1" CURR_VERSION = "3.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=f553650e-0093-48d6-9682-6f593964f5de TargetServiceName=null EntryTimestamp=2021-07-13T00:08:46.872Z InvocationID=8b58a4ff-b75a-45e5-9e5f-bc5c769e62d5 ErrorCode=500 During change state, another component 5387fb89-68a1-4ba7-b210-3145be9a83e8 has been created/updated 2021-07-13T00:11:40.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=472d1366-87d3-469d-a5d7-d51e61d02209 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:11:40.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=472d1366-87d3-469d-a5d7-d51e61d02209 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@23162014 2021-07-13T00:11:40.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=472d1366-87d3-469d-a5d7-d51e61d02209 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:11:40.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=472d1366-87d3-469d-a5d7-d51e61d02209 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:11:41.082Z [qtp817348612-96] WARN o.o.s.b.s.ResourceUploadServlet AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "fe6788fb-031f-4130-8053-e4104b13ac33" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "2.1" CURR_VERSION = "3.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=f553650e-0093-48d6-9682-6f593964f5de TargetServiceName=null EntryTimestamp=2021-07-13T00:08:46.872Z InvocationID=8b58a4ff-b75a-45e5-9e5f-bc5c769e62d5 ErrorCode=500 #uploadMultipart('multipart', /tmp/rep13474054655057363291tmp, 'form-data; filename="vduCp.zip"; name="resourceZip"', '{"payloadName": "vduCp.yml", "contactId": "jh0003", "name": "VDU Cp", "description": "VDU Cp", "resourceIconPath": "port", "resourceType": "CP", "categories": [{"name": "Generic", "subcategories": [{"name": "Network Elements"}]}], "vendorName": "ETSI", "vendorRelease": "4.1.1", "tags": ["VDU Cp"]}', 'Request(POST https://sdc-be.onap:8443/sdc2/rest/v1/catalog/upload/multipart?createNewVersion=true)@62228854', 'jh0003', true): org.apache.cxf.jaxrs.impl.ResponseImpl@4a4174c2 in 2.90min (too slow!) 2021-07-13T00:11:41.083Z [qtp817348612-96] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "CertificationSuccess" RESOURCE_NAME = "VDU Cp" RESOURCE_TYPE = "Resource" SERVICE_INSTANCE_ID = "fe6788fb-031f-4130-8053-e4104b13ac33" INVARIANT_UUID = "3db3a472-26d4-440d-9480-2d46aa9d9e4d" PREV_VERSION = "2.1" CURR_VERSION = "3.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "NOT_CERTIFIED_CHECKIN" CURR_STATE = "CERTIFIED" COMMENT = "certification on import" STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR ResponseCode=201 RequestID=null ElapsedTime=174210 PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp StatusCode=COMPLETE TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=f553650e-0093-48d6-9682-6f593964f5de TargetServiceName=null EntryTimestamp=2021-07-13T00:08:46.872Z InvocationID=8b58a4ff-b75a-45e5-9e5f-bc5c769e62d5 ErrorCode=500 LogTimestamp=2021-07-13T00:11:41.082982Z ResponseDesc=Created Exiting. 2021-07-13T00:11:41.084Z [qtp817348612-96] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 SC="201" 2021-07-13T00:11:41.164Z [qtp817348612-16] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=3798e7db-f7ec-4bda-988a-d46b80874d49 InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS RemoteHost=10.242.77.79 ServerIPAddress=10.242.98.212 ServiceName=v1/catalog/upload/multipart ServiceInstanceID=null EntryTimestamp=2021-07-13T00:11:41.164790Z InvocationID=434f57ab-21fe-48af-9181-cd32322314c0 LogTimestamp=2021-07-13T00:11:41.164852Z Entering 2021-07-13T00:11:41.165Z [qtp817348612-16] INFO o.o.sdc.be.filters.BeServletFilter RequestID=3798e7db-f7ec-4bda-988a-d46b80874d49 InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS userId=jh0003 RemoteHost=10.242.77.79 ServerIPAddress=10.242.98.212 ServiceName=v1/catalog/upload/multipart ServiceInstanceID=null EntryTimestamp=2021-07-13T00:11:41.164790Z InvocationID=434f57ab-21fe-48af-9181-cd32322314c0 LogTimestamp=2021-07-13T00:11:41.164852Z localAddr=10.242.98.212 remoteAddr=10.242.77.79 POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 2021-07-13T00:11:43.361Z [qtp817348612-16] INFO o.o.s.b.c.i.ComponentBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:11:41.165Z InvocationID=b87c7a34-340e-48bc-b94b-ec80912b3f67 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp LifecycleState cannot be defined by user. This field will be overridden by the application 2021-07-13T00:11:43.361Z [qtp817348612-16] INFO o.o.s.b.c.i.ComponentBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:11:41.165Z InvocationID=b87c7a34-340e-48bc-b94b-ec80912b3f67 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Version cannot be defined by user. This field will be overridden by the application 2021-07-13T00:11:43.361Z [qtp817348612-16] INFO o.o.s.b.c.i.ComponentBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:11:41.165Z InvocationID=b87c7a34-340e-48bc-b94b-ec80912b3f67 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Creator cannot be defined by user. This field will be overridden by the application 2021-07-13T00:11:43.361Z [qtp817348612-16] INFO o.o.s.b.c.i.ComponentBusinessLogic RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:11:41.165Z InvocationID=b87c7a34-340e-48bc-b94b-ec80912b3f67 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Is Highest Version cannot be defined by user. This field will be overridden by the application 2021-07-13T00:11:43.564Z [qtp817348612-16] INFO o.o.s.b.m.o.impl.GraphLockOperation RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true EntryTimestamp=2021-07-13T00:11:41.165Z InvocationID=b87c7a34-340e-48bc-b94b-ec80912b3f67 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp lock resource with name VirtualCp 2021-07-13T00:11:43.977Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=472d1366-87d3-469d-a5d7-d51e61d02209 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out 2021-07-13T00:11:43.977Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=472d1366-87d3-469d-a5d7-d51e61d02209 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:11:45.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a242fe52-aa1f-4543-8c02-a453b1a69dab PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:11:45.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a242fe52-aa1f-4543-8c02-a453b1a69dab PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3849ad88 2021-07-13T00:11:45.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a242fe52-aa1f-4543-8c02-a453b1a69dab PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:11:45.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a242fe52-aa1f-4543-8c02-a453b1a69dab PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:11:48.667Z [qtp817348612-16] WARN o.j.d.l.c.ConsistentKeyLocker AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "Virtual Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "250a7591-b309-4208-997a-6fa251d5f51b" INVARIANT_UUID = "716c15a0-9ce4-4419-be1e-ba9bccf8f84b" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "" CURR_STATE = "CERTIFIED" STATUS = "201" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.VirtualCp" RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=250a7591-b309-4208-997a-6fa251d5f51b EntryTimestamp=2021-07-13T00:11:41.165Z InvocationID=b87c7a34-340e-48bc-b94b-ec80912b3f67 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Temporary storage failure while checking locks org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.getSliceWithRetries(ConsistentKeyLocker.java:464) at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSingleLock(ConsistentKeyLocker.java:426) at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSingleLock(ConsistentKeyLocker.java:118) at org.janusgraph.diskstorage.locking.AbstractLocker.checkLocks(AbstractLocker.java:351) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.checkAllLocks(ExpectedValueCheckingTransaction.java:175) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.prepareForMutations(ExpectedValueCheckingTransaction.java:154) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStoreManager.mutateMany(ExpectedValueCheckingStoreManager.java:72) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:94) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:91) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:91) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:139) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:196) at org.janusgraph.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:134) at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:733) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1379) at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.doCommit(JanusGraphBlueprintsGraph.java:298) at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:104) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.commit(JanusGraphClient.java:250) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.unlockNode(JanusGraphGenericDao.java:1272) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.releaseElement(JanusGraphGenericDao.java:1284) at org.openecomp.sdc.be.model.operations.impl.GraphLockOperation.unlockComponentByName(GraphLockOperation.java:71) at org.openecomp.sdc.be.components.impl.ResourceBusinessLogic.createResourceByDao(ResourceBusinessLogic.java:3765) at org.openecomp.sdc.be.components.impl.ResourceBusinessLogic.createResourceByImport(ResourceBusinessLogic.java:3364) at org.openecomp.sdc.be.components.impl.ResourceBusinessLogic.createOrUpdateResourceByImport(ResourceBusinessLogic.java:3319) at org.openecomp.sdc.be.components.impl.ResourceImportManager.importCertifiedResource(ResourceImportManager.java:188) at org.openecomp.sdc.be.components.impl.ResourceImportManager.importNormativeResource(ResourceImportManager.java:143) at org.openecomp.sdc.be.servlets.AbstractValidationsServlet.handleImport(AbstractValidationsServlet.java:659) at org.openecomp.sdc.be.servlets.ResourceUploadServlet.uploadMultipart_aroundBody0(ResourceUploadServlet.java:131) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$AjcClosure1.run(ResourceUploadServlet.java:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:212) at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) at com.jcabi.aspects.aj.MethodLogger.wrapClass(MethodLogger.java:136) at org.openecomp.sdc.be.servlets.ResourceUploadServlet.uploadMultipart(ResourceUploadServlet.java:109) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$$FastClassBySpringCGLIB$$6732af97.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$$EnhancerBySpringCGLIB$$66c7ac05.uploadMultipart() at jdk.internal.reflect.GeneratedMethodAccessor368.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1411) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:226) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.togglz.servlet.TogglzFilter.doFilter(TogglzFilter.java:100) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.ReqValidationFilter.doFilter(ReqValidationFilter.java:72) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.GatewayFilter.doFilter(GatewayFilter.java:71) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1630) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:501) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ALL (3 responses were required but only 2 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ALL (3 responses were required but only 2 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ALL (3 responses were required but only 2 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:11:48.690Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=a242fe52-aa1f-4543-8c02-a453b1a69dab ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out 2021-07-13T00:11:48.691Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=a242fe52-aa1f-4543-8c02-a453b1a69dab ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:11:49.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af64417e-9e1a-40e0-b702-d0d67e80eda8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:11:49.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af64417e-9e1a-40e0-b702-d0d67e80eda8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1dba4e00 2021-07-13T00:11:49.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af64417e-9e1a-40e0-b702-d0d67e80eda8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:11:49.571Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af64417e-9e1a-40e0-b702-d0d67e80eda8 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:11:49.666Z [qtp817348612-16] INFO o.o.s.b.c.d.e.DmaapClientFactory AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "Virtual Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "250a7591-b309-4208-997a-6fa251d5f51b" INVARIANT_UUID = "716c15a0-9ce4-4419-be1e-ba9bccf8f84b" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "" CURR_STATE = "CERTIFIED" STATUS = "201" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.VirtualCp" RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=250a7591-b309-4208-997a-6fa251d5f51b EntryTimestamp=2021-07-13T00:11:41.165Z InvocationID=b87c7a34-340e-48bc-b94b-ec80912b3f67 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp The DmaapProducerConfiguration is DmaapProducerConfiguration(active=true, hosts=olsd004.com:3905, consumerGroup=sdc-AUTO-1626130692, consumerId=sdc-AUTO1-1626130692, timeoutMs=15000, limit=1, pollingInterval=2, topic=SDC-FACADE-NOTIF-v1, latitude=32.109333, longitude=34.855499, version=1.0, serviceName=dmaap.com:3905/events, environment=TEST, partner=BOT_R, routeOffer=MR1, protocol=null, contenttype=application/json, dme2TraceOn=true, aftEnvironment=null, aftDme2ConnectionTimeoutMs=15000, aftDme2RoundtripTimeoutMs=240000, aftDme2ReadTimeoutMs=50000, dme2preferredRouterFilePath=null, credential=Credential [username=user1@sdc.com, password=password==], timeLimitForNotificationHandleMs=120000, aftDme2SslEnable=true, aftDme2ClientIgnoreSslConfig=false, aftDme2ClientKeystore=/app/jetty/etc/truststore, aftDme2ClientKeystorePassword=dmaap_truststore_pwd, aftDme2ClientSslCertAlias=certman) 2021-07-13T00:11:49.667Z [qtp817348612-16] ERROR o.o.s.be.catalog.impl.DmaapProducer AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "Virtual Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "250a7591-b309-4208-997a-6fa251d5f51b" INVARIANT_UUID = "716c15a0-9ce4-4419-be1e-ba9bccf8f84b" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "" CURR_STATE = "CERTIFIED" STATUS = "201" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.VirtualCp" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=250a7591-b309-4208-997a-6fa251d5f51b TargetServiceName=null EntryTimestamp=2021-07-13T00:11:41.165Z InvocationID=b87c7a34-340e-48bc-b94b-ec80912b3f67 ErrorCode=500 Failed to connect to topic . Exeption Input byte array has wrong 4-byte ending unit 2021-07-13T00:11:52.587Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=af64417e-9e1a-40e0-b702-d0d67e80eda8 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out 2021-07-13T00:11:52.588Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=af64417e-9e1a-40e0-b702-d0d67e80eda8 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:11:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:11:52.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:11:55.222Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d08aeb77-6314-4915-b37d-d6bd4eefea5f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:11:55.222Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d08aeb77-6314-4915-b37d-d6bd4eefea5f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5c576cb8 2021-07-13T00:11:55.222Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d08aeb77-6314-4915-b37d-d6bd4eefea5f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:11:55.222Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d08aeb77-6314-4915-b37d-d6bd4eefea5f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:11:56.472Z [qtp817348612-16] INFO o.o.s.b.m.o.impl.GraphLockOperation AuditMessage=ACTION = "ResourceImport" RESOURCE_NAME = "Virtual Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "250a7591-b309-4208-997a-6fa251d5f51b" INVARIANT_UUID = "716c15a0-9ce4-4419-be1e-ba9bccf8f84b" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "Jimmy Hendrix(jh0003)" PREV_STATE = "" CURR_STATE = "CERTIFIED" STATUS = "201" DESC = "OK" TOSCA_NODE_TYPE = "tosca.nodes.nfv.VirtualCp" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=250a7591-b309-4208-997a-6fa251d5f51b TargetServiceName=null EntryTimestamp=2021-07-13T00:11:41.165Z InvocationID=b87c7a34-340e-48bc-b94b-ec80912b3f67 ErrorCode=500 lock resource with id b5a27820-7c36-4011-b27b-a882f0b93fad 2021-07-13T00:11:58.236Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=d08aeb77-6314-4915-b37d-d6bd4eefea5f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out 2021-07-13T00:11:58.237Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=d08aeb77-6314-4915-b37d-d6bd4eefea5f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:11:59.626Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ca3a36c-fac2-449b-a4c9-513b1b391f1f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:11:59.626Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ca3a36c-fac2-449b-a4c9-513b1b391f1f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6ade43e5 2021-07-13T00:11:59.626Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ca3a36c-fac2-449b-a4c9-513b1b391f1f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:11:59.626Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ca3a36c-fac2-449b-a4c9-513b1b391f1f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:12:01.123Z [qtp817348612-16] WARN o.j.d.l.c.ConsistentKeyLocker AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "Virtual Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "250a7591-b309-4208-997a-6fa251d5f51b" INVARIANT_UUID = "716c15a0-9ce4-4419-be1e-ba9bccf8f84b" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "b5a27820-7c36-4011-b27b-a882f0b93fad.assettoscatemplate" CURR_ARTIFACT_UUID = "b5a27820-7c36-4011-b27b-a882f0b93fad.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VirtualCp-template.yml,0,b5a27820-7c36-4011-b27b-a882f0b93fad.assettoscatemplate,0 " STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=f3e6142a-11d1-44bd-989d-d1ddb2c2c288 TargetServiceName=null EntryTimestamp=2021-07-13T00:11:41.165Z InvocationID=b87c7a34-340e-48bc-b94b-ec80912b3f67 ErrorCode=500 Temporary storage failure while checking locks org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.getSliceWithRetries(ConsistentKeyLocker.java:464) at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSingleLock(ConsistentKeyLocker.java:426) at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSingleLock(ConsistentKeyLocker.java:118) at org.janusgraph.diskstorage.locking.AbstractLocker.checkLocks(AbstractLocker.java:351) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.checkAllLocks(ExpectedValueCheckingTransaction.java:175) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.prepareForMutations(ExpectedValueCheckingTransaction.java:154) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStoreManager.mutateMany(ExpectedValueCheckingStoreManager.java:72) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:94) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:91) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:91) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:139) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:196) at org.janusgraph.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:134) at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:733) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1379) at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.doCommit(JanusGraphBlueprintsGraph.java:298) at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:104) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.commit(JanusGraphClient.java:250) at org.openecomp.sdc.be.dao.jsongraph.JanusGraphDao.commit(JanusGraphDao.java:76) at org.openecomp.sdc.be.components.impl.BaseBusinessLogic.unlockComponent(BaseBusinessLogic.java:286) at org.openecomp.sdc.be.components.impl.ArtifactsBusinessLogic.lockComponentAndUpdateArtifact(ArtifactsBusinessLogic.java:877) at org.openecomp.sdc.be.components.impl.ArtifactsBusinessLogic.lambda$generateAndSaveToscaArtifact$3(ArtifactsBusinessLogic.java:355) at fj.data.Either$LeftProjection.bind(Either.java:287) at org.openecomp.sdc.be.components.impl.ArtifactsBusinessLogic.generateAndSaveToscaArtifact(ArtifactsBusinessLogic.java:350) at org.openecomp.sdc.be.components.impl.ComponentBusinessLogic.saveToscaArtifactPayload(ComponentBusinessLogic.java:515) at org.openecomp.sdc.be.components.impl.ComponentBusinessLogic.lambda$populateToscaArtifacts$3(ComponentBusinessLogic.java:496) at org.openecomp.sdc.be.components.impl.ComponentBusinessLogic.saveToscaArtifactAndPopulateToscaArtifactsWithResult(ComponentBusinessLogic.java:145) at org.openecomp.sdc.be.components.impl.ComponentBusinessLogic.populateToscaArtifacts(ComponentBusinessLogic.java:494) at org.openecomp.sdc.be.components.impl.ComponentBusinessLogic.populateToscaArtifacts(ComponentBusinessLogic.java:480) at org.openecomp.sdc.be.components.impl.ResourceBusinessLogic.propagateStateToCertified(ResourceBusinessLogic.java:3258) at org.openecomp.sdc.be.components.impl.ResourceImportManager.importCertifiedResource(ResourceImportManager.java:200) at org.openecomp.sdc.be.components.impl.ResourceImportManager.importNormativeResource(ResourceImportManager.java:143) at org.openecomp.sdc.be.servlets.AbstractValidationsServlet.handleImport(AbstractValidationsServlet.java:659) at org.openecomp.sdc.be.servlets.ResourceUploadServlet.uploadMultipart_aroundBody0(ResourceUploadServlet.java:131) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$AjcClosure1.run(ResourceUploadServlet.java:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:212) at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) at com.jcabi.aspects.aj.MethodLogger.wrapClass(MethodLogger.java:136) at org.openecomp.sdc.be.servlets.ResourceUploadServlet.uploadMultipart(ResourceUploadServlet.java:109) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$$FastClassBySpringCGLIB$$6732af97.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$$EnhancerBySpringCGLIB$$66c7ac05.uploadMultipart() at jdk.internal.reflect.GeneratedMethodAccessor368.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1411) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:226) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.togglz.servlet.TogglzFilter.doFilter(TogglzFilter.java:100) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.ReqValidationFilter.doFilter(ReqValidationFilter.java:72) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.GatewayFilter.doFilter(GatewayFilter.java:71) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1630) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:501) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ALL (3 responses were required but only 2 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ALL (3 responses were required but only 2 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ALL (3 responses were required but only 2 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:12:02.643Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=5ca3a36c-fac2-449b-a4c9-513b1b391f1f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out 2021-07-13T00:12:02.643Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=5ca3a36c-fac2-449b-a4c9-513b1b391f1f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:12:06.710Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0828ef01-2ba8-4ecc-92cc-54b8402a8b6a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:12:06.710Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0828ef01-2ba8-4ecc-92cc-54b8402a8b6a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@19971ed6 2021-07-13T00:12:06.710Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0828ef01-2ba8-4ecc-92cc-54b8402a8b6a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:12:06.710Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0828ef01-2ba8-4ecc-92cc-54b8402a8b6a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:12:07.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:12:07.919Z [qtp817348612-16] WARN o.j.d.l.c.ConsistentKeyLocker AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "Virtual Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "250a7591-b309-4208-997a-6fa251d5f51b" INVARIANT_UUID = "716c15a0-9ce4-4419-be1e-ba9bccf8f84b" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "b5a27820-7c36-4011-b27b-a882f0b93fad.assettoscatemplate" CURR_ARTIFACT_UUID = "b5a27820-7c36-4011-b27b-a882f0b93fad.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VirtualCp-template.yml,0,b5a27820-7c36-4011-b27b-a882f0b93fad.assettoscatemplate,0 " STATUS = "200" DESC = "OK" RequestID=null PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=f3e6142a-11d1-44bd-989d-d1ddb2c2c288 TargetServiceName=null EntryTimestamp=2021-07-13T00:11:41.165Z InvocationID=b87c7a34-340e-48bc-b94b-ec80912b3f67 ErrorCode=500 Temporary storage failure while checking locks org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.getSliceWithRetries(ConsistentKeyLocker.java:464) at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSingleLock(ConsistentKeyLocker.java:426) at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSingleLock(ConsistentKeyLocker.java:118) at org.janusgraph.diskstorage.locking.AbstractLocker.checkLocks(AbstractLocker.java:351) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.checkAllLocks(ExpectedValueCheckingTransaction.java:175) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.prepareForMutations(ExpectedValueCheckingTransaction.java:154) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStoreManager.mutateMany(ExpectedValueCheckingStoreManager.java:72) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:94) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:91) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:91) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:139) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:196) at org.janusgraph.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:134) at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:733) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1379) at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.doCommit(JanusGraphBlueprintsGraph.java:298) at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:104) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.commit(JanusGraphClient.java:250) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.unlockNode(JanusGraphGenericDao.java:1272) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.releaseElement(JanusGraphGenericDao.java:1284) at org.openecomp.sdc.be.model.operations.impl.GraphLockOperation.unlockComponent(GraphLockOperation.java:65) at org.openecomp.sdc.be.components.impl.BaseBusinessLogic.unlockComponent(BaseBusinessLogic.java:290) at org.openecomp.sdc.be.components.impl.ArtifactsBusinessLogic.lockComponentAndUpdateArtifact(ArtifactsBusinessLogic.java:877) at org.openecomp.sdc.be.components.impl.ArtifactsBusinessLogic.lambda$generateAndSaveToscaArtifact$3(ArtifactsBusinessLogic.java:355) at fj.data.Either$LeftProjection.bind(Either.java:287) at org.openecomp.sdc.be.components.impl.ArtifactsBusinessLogic.generateAndSaveToscaArtifact(ArtifactsBusinessLogic.java:350) at org.openecomp.sdc.be.components.impl.ComponentBusinessLogic.saveToscaArtifactPayload(ComponentBusinessLogic.java:515) at org.openecomp.sdc.be.components.impl.ComponentBusinessLogic.lambda$populateToscaArtifacts$3(ComponentBusinessLogic.java:496) at org.openecomp.sdc.be.components.impl.ComponentBusinessLogic.saveToscaArtifactAndPopulateToscaArtifactsWithResult(ComponentBusinessLogic.java:145) at org.openecomp.sdc.be.components.impl.ComponentBusinessLogic.populateToscaArtifacts(ComponentBusinessLogic.java:494) at org.openecomp.sdc.be.components.impl.ComponentBusinessLogic.populateToscaArtifacts(ComponentBusinessLogic.java:480) at org.openecomp.sdc.be.components.impl.ResourceBusinessLogic.propagateStateToCertified(ResourceBusinessLogic.java:3258) at org.openecomp.sdc.be.components.impl.ResourceImportManager.importCertifiedResource(ResourceImportManager.java:200) at org.openecomp.sdc.be.components.impl.ResourceImportManager.importNormativeResource(ResourceImportManager.java:143) at org.openecomp.sdc.be.servlets.AbstractValidationsServlet.handleImport(AbstractValidationsServlet.java:659) at org.openecomp.sdc.be.servlets.ResourceUploadServlet.uploadMultipart_aroundBody0(ResourceUploadServlet.java:131) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$AjcClosure1.run(ResourceUploadServlet.java:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:212) at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) at com.jcabi.aspects.aj.MethodLogger.wrapClass(MethodLogger.java:136) at org.openecomp.sdc.be.servlets.ResourceUploadServlet.uploadMultipart(ResourceUploadServlet.java:109) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$$FastClassBySpringCGLIB$$6732af97.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at org.openecomp.sdc.be.servlets.ResourceUploadServlet$$EnhancerBySpringCGLIB$$66c7ac05.uploadMultipart() at jdk.internal.reflect.GeneratedMethodAccessor368.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1411) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:226) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.togglz.servlet.TogglzFilter.doFilter(TogglzFilter.java:100) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.ReqValidationFilter.doFilter(ReqValidationFilter.java:72) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.GatewayFilter.doFilter(GatewayFilter.java:71) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1630) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:501) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ALL (3 responses were required but only 2 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ALL (3 responses were required but only 2 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ALL (3 responses were required but only 2 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:12:07.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:12:08.423Z [qtp817348612-16] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "Virtual Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "250a7591-b309-4208-997a-6fa251d5f51b" INVARIANT_UUID = "716c15a0-9ce4-4419-be1e-ba9bccf8f84b" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "b5a27820-7c36-4011-b27b-a882f0b93fad.assettoscatemplate" CURR_ARTIFACT_UUID = "b5a27820-7c36-4011-b27b-a882f0b93fad.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VirtualCp-template.yml,0,b5a27820-7c36-4011-b27b-a882f0b93fad.assettoscatemplate,0 " STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=STARTED ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=f3e6142a-11d1-44bd-989d-d1ddb2c2c288 TargetServiceName=null EntryTimestamp=2021-07-13T00:11:41.165Z InvocationID=b87c7a34-340e-48bc-b94b-ec80912b3f67 ErrorCode=500 Starting to create Csar for component Virtual Cp 2021-07-13T00:12:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:12:09.367Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:12:09.367Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:12:09.789Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=0828ef01-2ba8-4ecc-92cc-54b8402a8b6a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:12:09.789Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=0828ef01-2ba8-4ecc-92cc-54b8402a8b6a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:12:12.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1eead355-b4d3-4f6b-a087-493a413c1868 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:12:12.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1eead355-b4d3-4f6b-a087-493a413c1868 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7a762bec 2021-07-13T00:12:12.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1eead355-b4d3-4f6b-a087-493a413c1868 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:12:12.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1eead355-b4d3-4f6b-a087-493a413c1868 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:12:13.106Z [qtp817348612-16] INFO org.openecomp.sdc.be.tosca.CsarUtils AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "Virtual Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "250a7591-b309-4208-997a-6fa251d5f51b" INVARIANT_UUID = "716c15a0-9ce4-4419-be1e-ba9bccf8f84b" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "b5a27820-7c36-4011-b27b-a882f0b93fad.assettoscatemplate" CURR_ARTIFACT_UUID = "b5a27820-7c36-4011-b27b-a882f0b93fad.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VirtualCp-template.yml,0,b5a27820-7c36-4011-b27b-a882f0b93fad.assettoscatemplate,0 " STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=f3e6142a-11d1-44bd-989d-d1ddb2c2c288 TargetServiceName=null EntryTimestamp=2021-07-13T00:11:41.165Z InvocationID=b87c7a34-340e-48bc-b94b-ec80912b3f67 ErrorCode=500 Ended create Csar for component Virtual Cp 2021-07-13T00:12:13.107Z [qtp817348612-16] INFO o.o.s.b.m.o.impl.GraphLockOperation AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "Virtual Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "250a7591-b309-4208-997a-6fa251d5f51b" INVARIANT_UUID = "716c15a0-9ce4-4419-be1e-ba9bccf8f84b" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "b5a27820-7c36-4011-b27b-a882f0b93fad.assettoscatemplate" CURR_ARTIFACT_UUID = "b5a27820-7c36-4011-b27b-a882f0b93fad.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VirtualCp-template.yml,0,b5a27820-7c36-4011-b27b-a882f0b93fad.assettoscatemplate,0 " STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=f3e6142a-11d1-44bd-989d-d1ddb2c2c288 TargetServiceName=null EntryTimestamp=2021-07-13T00:11:41.165Z InvocationID=b87c7a34-340e-48bc-b94b-ec80912b3f67 ErrorCode=500 lock resource with id b5a27820-7c36-4011-b27b-a882f0b93fad 2021-07-13T00:12:13.811Z [qtp817348612-16] WARN o.j.d.l.c.ConsistentKeyLocker AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "Virtual Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "250a7591-b309-4208-997a-6fa251d5f51b" INVARIANT_UUID = "716c15a0-9ce4-4419-be1e-ba9bccf8f84b" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "b5a27820-7c36-4011-b27b-a882f0b93fad.assettoscatemplate" CURR_ARTIFACT_UUID = "b5a27820-7c36-4011-b27b-a882f0b93fad.assettoscatemplate" ARTIFACT_DATA = "TOSCA,'assettoscatemplate',TOSCA_TEMPLATE,resource-VirtualCp-template.yml,0,b5a27820-7c36-4011-b27b-a882f0b93fad.assettoscatemplate,0 " STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR RequestID=null PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=f3e6142a-11d1-44bd-989d-d1ddb2c2c288 TargetServiceName=null EntryTimestamp=2021-07-13T00:11:41.165Z InvocationID=b87c7a34-340e-48bc-b94b-ec80912b3f67 ErrorCode=500 Lock write succeeded but took too long: duration PT0.651114S exceeded limit PT0.5S 2021-07-13T00:12:15.806Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=1eead355-b4d3-4f6b-a087-493a413c1868 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:12:15.806Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=1eead355-b4d3-4f6b-a087-493a413c1868 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:12:18.619Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a79a695-5196-4445-b7e2-f035c258c3f8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:12:18.619Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a79a695-5196-4445-b7e2-f035c258c3f8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@31e68c9b 2021-07-13T00:12:18.619Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a79a695-5196-4445-b7e2-f035c258c3f8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:12:18.619Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a79a695-5196-4445-b7e2-f035c258c3f8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:12:19.167Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=2a79a695-5196-4445-b7e2-f035c258c3f8 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:12:19.167Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=2a79a695-5196-4445-b7e2-f035c258c3f8 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:12:19.673Z [qtp817348612-16] INFO o.o.l.f.base.AbstractAuditLogFilter AuditMessage=ACTION = "ArtifactPayloadUpdate" RESOURCE_NAME = "Virtual Cp" RESOURCE_TYPE = "CP" SERVICE_INSTANCE_ID = "250a7591-b309-4208-997a-6fa251d5f51b" INVARIANT_UUID = "716c15a0-9ce4-4419-be1e-ba9bccf8f84b" PREV_VERSION = "" CURR_VERSION = "1.0" MODIFIER = "(UNKNOWN)" PREV_STATE = "" CURR_STATE = "CERTIFIED" PREV_ARTIFACT_UUID = "b5a27820-7c36-4011-b27b-a882f0b93fad.assettoscacsar" CURR_ARTIFACT_UUID = "b5a27820-7c36-4011-b27b-a882f0b93fad.assettoscacsar" ARTIFACT_DATA = "TOSCA,'assettoscacsar',TOSCA_CSAR,resource-VirtualCp-csar.csar,0,b5a27820-7c36-4011-b27b-a882f0b93fad.assettoscacsar,0 " STATUS = "200" DESC = "OK" SupportablityAction=GENERATE CSAR ResponseCode=201 RequestID=null ElapsedTime=38507 PartnerName=jh0003 auditOn=true SupportablityStatus=COMPLETE ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp StatusCode=COMPLETE TargetEntity=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/upload/multipart?createNewVersion=true ServiceInstanceID=a247f57c-8048-4748-842d-e501ac6e4617 TargetServiceName=null EntryTimestamp=2021-07-13T00:11:41.165Z InvocationID=b87c7a34-340e-48bc-b94b-ec80912b3f67 ErrorCode=500 LogTimestamp=2021-07-13T00:12:19.672579Z ResponseDesc=Created Exiting. 2021-07-13T00:12:19.673Z [qtp817348612-16] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc2/rest/v1/catalog/upload/multipart HTTP/1.1 SC="201" 2021-07-13T00:12:19.866Z [qtp817348612-101] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=7251ede0-054c-4a92-ae95-4f07e1303c50 InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS RemoteHost=10.242.77.79 ServerIPAddress=10.242.98.212 ServiceName=v1/catalog/uploadType/grouptypes ServiceInstanceID=null EntryTimestamp=2021-07-13T00:12:19.866218Z InvocationID=126c2f76-09dd-457c-a43c-f9d055fec7d9 LogTimestamp=2021-07-13T00:12:19.866287Z Entering 2021-07-13T00:12:19.866Z [qtp817348612-101] INFO o.o.sdc.be.filters.BeServletFilter RequestID=7251ede0-054c-4a92-ae95-4f07e1303c50 InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS userId=jh0003 RemoteHost=10.242.77.79 ServerIPAddress=10.242.98.212 ServiceName=v1/catalog/uploadType/grouptypes ServiceInstanceID=null EntryTimestamp=2021-07-13T00:12:19.866218Z InvocationID=126c2f76-09dd-457c-a43c-f9d055fec7d9 LogTimestamp=2021-07-13T00:12:19.866287Z localAddr=10.242.98.212 remoteAddr=10.242.77.79 POST /sdc2/rest/v1/catalog/uploadType/grouptypes HTTP/1.1 2021-07-13T00:12:19.961Z [qtp817348612-101] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/grouptypes EntryTimestamp=2021-07-13T00:12:19.867Z InvocationID=a9635568-78a8-46b6-b86a-a81c6c592e9f PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send GROUP_TYPE : tosca.groups.Root to dao for create 2021-07-13T00:12:20.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eaf1f7b7-0264-488c-b3bf-5b95e80dd57b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:12:20.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eaf1f7b7-0264-488c-b3bf-5b95e80dd57b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64dc1e65 2021-07-13T00:12:20.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eaf1f7b7-0264-488c-b3bf-5b95e80dd57b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:12:20.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eaf1f7b7-0264-488c-b3bf-5b95e80dd57b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:12:20.761Z [qtp817348612-101] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/grouptypes EntryTimestamp=2021-07-13T00:12:19.867Z InvocationID=a9635568-78a8-46b6-b86a-a81c6c592e9f PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send GROUP_TYPE : org.openecomp.groups.heat.HeatStack to dao for create 2021-07-13T00:12:21.560Z [qtp817348612-101] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/grouptypes EntryTimestamp=2021-07-13T00:12:19.867Z InvocationID=a9635568-78a8-46b6-b86a-a81c6c592e9f PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send GROUP_TYPE : org.openecomp.groups.VfModule to dao for create 2021-07-13T00:12:22.260Z [qtp817348612-101] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/grouptypes EntryTimestamp=2021-07-13T00:12:19.867Z InvocationID=a9635568-78a8-46b6-b86a-a81c6c592e9f PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send GROUP_TYPE : org.openecomp.groups.NetworkCollection to dao for create 2021-07-13T00:12:22.374Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11f8a0dd-0f78-497c-ae2e-0e9fc0f94ece PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:12:22.374Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11f8a0dd-0f78-497c-ae2e-0e9fc0f94ece PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@465742f0 2021-07-13T00:12:22.374Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11f8a0dd-0f78-497c-ae2e-0e9fc0f94ece PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:12:22.374Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11f8a0dd-0f78-497c-ae2e-0e9fc0f94ece PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:12:22.460Z [qtp817348612-101] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/grouptypes EntryTimestamp=2021-07-13T00:12:19.867Z InvocationID=a9635568-78a8-46b6-b86a-a81c6c592e9f PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send GROUP_TYPE : org.openecomp.groups.VfcInstanceGroup to dao for create 2021-07-13T00:12:22.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:12:22.928Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:12:24.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7702d508-6d15-4c77-8e5f-83b28eaa2db2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:12:24.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7702d508-6d15-4c77-8e5f-83b28eaa2db2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@43d09cf5 2021-07-13T00:12:24.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7702d508-6d15-4c77-8e5f-83b28eaa2db2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:12:24.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7702d508-6d15-4c77-8e5f-83b28eaa2db2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:12:24.993Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:12:26.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c25fc4b6-2433-4796-bd21-767eeefc370a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:12:26.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c25fc4b6-2433-4796-bd21-767eeefc370a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f7b326b 2021-07-13T00:12:26.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c25fc4b6-2433-4796-bd21-767eeefc370a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:12:26.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c25fc4b6-2433-4796-bd21-767eeefc370a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:12:28.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a57a9bb-5358-4fbd-a170-2fcfed6e546d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:12:28.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a57a9bb-5358-4fbd-a170-2fcfed6e546d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1bd038d7 2021-07-13T00:12:28.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a57a9bb-5358-4fbd-a170-2fcfed6e546d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:12:28.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a57a9bb-5358-4fbd-a170-2fcfed6e546d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:12:30.966Z [qtp817348612-101] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/grouptypes EntryTimestamp=2021-07-13T00:12:19.867Z InvocationID=a9635568-78a8-46b6-b86a-a81c6c592e9f PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send GROUP_TYPE : tosca.groups.nfv.PlacementGroup to dao for create 2021-07-13T00:12:31.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89dcff3a-75a5-48f5-8d8e-a45577ed3474 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:12:31.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89dcff3a-75a5-48f5-8d8e-a45577ed3474 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@676cb99f 2021-07-13T00:12:31.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89dcff3a-75a5-48f5-8d8e-a45577ed3474 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:12:31.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89dcff3a-75a5-48f5-8d8e-a45577ed3474 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:12:32.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32a62769-09b4-4c88-acf8-1878e061d0c7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:12:32.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32a62769-09b4-4c88-acf8-1878e061d0c7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@209ca6b3 2021-07-13T00:12:32.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32a62769-09b4-4c88-acf8-1878e061d0c7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:12:32.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32a62769-09b4-4c88-acf8-1878e061d0c7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:12:34.812Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a54b8e9b-2545-4c9a-aac3-e423f8c13b01 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:12:34.812Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a54b8e9b-2545-4c9a-aac3-e423f8c13b01 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@663220a1 2021-07-13T00:12:34.812Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a54b8e9b-2545-4c9a-aac3-e423f8c13b01 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:12:34.812Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a54b8e9b-2545-4c9a-aac3-e423f8c13b01 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:12:36.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9613a9d8-3241-4bb8-8b64-7a3a628480fe PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:12:36.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9613a9d8-3241-4bb8-8b64-7a3a628480fe PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7cbeba30 2021-07-13T00:12:36.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9613a9d8-3241-4bb8-8b64-7a3a628480fe PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:12:36.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9613a9d8-3241-4bb8-8b64-7a3a628480fe PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:12:37.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:12:37.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:12:38.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b39569c5-9bac-418b-be9d-b31c44d6b75a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:12:38.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b39569c5-9bac-418b-be9d-b31c44d6b75a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7e9c4479 2021-07-13T00:12:38.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b39569c5-9bac-418b-be9d-b31c44d6b75a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:12:38.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b39569c5-9bac-418b-be9d-b31c44d6b75a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:12:38.863Z [qtp817348612-101] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/grouptypes EntryTimestamp=2021-07-13T00:12:19.867Z InvocationID=a9635568-78a8-46b6-b86a-a81c6c592e9f PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send GROUP_TYPE : tosca.groups.nfv.NsPlacementGroup to dao for create 2021-07-13T00:12:40.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=934c24cf-479e-4ea9-9a17-156156be4d3f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:12:40.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=934c24cf-479e-4ea9-9a17-156156be4d3f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3bf0dc27 2021-07-13T00:12:40.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=934c24cf-479e-4ea9-9a17-156156be4d3f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:12:40.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=934c24cf-479e-4ea9-9a17-156156be4d3f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:12:41.770Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5dda88b2-c1d9-4b2c-8dc1-93acb3014c83 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:12:41.770Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5dda88b2-c1d9-4b2c-8dc1-93acb3014c83 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4c30d4d0 2021-07-13T00:12:41.770Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5dda88b2-c1d9-4b2c-8dc1-93acb3014c83 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:12:41.770Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5dda88b2-c1d9-4b2c-8dc1-93acb3014c83 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:12:42.860Z [qtp817348612-101] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/grouptypes EntryTimestamp=2021-07-13T00:12:19.867Z InvocationID=a9635568-78a8-46b6-b86a-a81c6c592e9f PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send GROUP_TYPE : tosca.groups.nfv.VNFFG to dao for create 2021-07-13T00:12:43.120Z [qtp817348612-101] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/grouptypes EntryTimestamp=2021-07-13T00:12:19.867Z InvocationID=a9635568-78a8-46b6-b86a-a81c6c592e9f PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp all GroupTypeDataDefinition [uniqueId=tosca.groups.nfv.VNFFG.1.0.grouptype, type=tosca.groups.nfv.VNFFG, name=NFV VNF Forwarding Graph, icon=group-nfv-placementgroup, version=1.0, members=[tosca.nodes.nfv.NFP, tosca.nodes.nfv.VNF, tosca.nodes.nfv.NS, tosca.nodes.nfv.NsVirtualLink, tosca.nodes.nfv.NfpPositionElement], metadata=null, description=the VNFFG group type describes a topology of the NS or a portion of the NS, and optionally forwarding rules, applicable to the traffic conveyed over this topology, creationTime=1626135162867, modificationTime=1626135162867] [properties=[PropertyDefinition [ PropertyDataDefinition(definition=false, hidden=false, uniqueId=tosca.groups.nfv.VNFFG.1.0.grouptype.description, type=string, required=true, defaultValue=null, description=Human readable description of the group, schema=null, password=false, name=description, value=null, label=null, immutable=false, mappedToComponentProperty=true, toscaGetFunctionType=null, inputPath=null, status=null, inputId=null, instanceUniqueId=null, propertyId=null, parentPropertyType=null, subPropertyInputPath=null, annotations=null, parentUniqueId=null, getInputValues=null, isDeclaredListInput=false, getPolicyValues=null, propertyConstraints=null, metadata=null), name=description, constraints=null]]], capabilities={}] were created successfully!!! 2021-07-13T00:12:44.920Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eab92147-47ec-4dc3-9737-aac437f9aed7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:12:44.920Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eab92147-47ec-4dc3-9737-aac437f9aed7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4556885c 2021-07-13T00:12:44.920Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eab92147-47ec-4dc3-9737-aac437f9aed7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:12:44.920Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eab92147-47ec-4dc3-9737-aac437f9aed7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:12:46.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f65591a-5961-461f-83a8-bfc1f9a6ba9b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:12:46.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f65591a-5961-461f-83a8-bfc1f9a6ba9b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5975b303 2021-07-13T00:12:46.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f65591a-5961-461f-83a8-bfc1f9a6ba9b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:12:46.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f65591a-5961-461f-83a8-bfc1f9a6ba9b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:12:49.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:12:49.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:12:49.464Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=6f65591a-5961-461f-83a8-bfc1f9a6ba9b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:12:49.465Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=6f65591a-5961-461f-83a8-bfc1f9a6ba9b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:12:49.465Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6f65591a-5961-461f-83a8-bfc1f9a6ba9b ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:12:49.465Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6f65591a-5961-461f-83a8-bfc1f9a6ba9b ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:12:52.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:12:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:12:52.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:12:55.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:12:58.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:12:58.761Z [qtp817348612-101] WARN o.j.d.l.c.ConsistentKeyLocker RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/grouptypes EntryTimestamp=2021-07-13T00:12:19.867Z InvocationID=a9635568-78a8-46b6-b86a-a81c6c592e9f PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Lock write succeeded but took too long: duration PT12.093823S exceeded limit PT0.5S 2021-07-13T00:13:01.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:13:04.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:13:07.011Z [qtp817348612-101] WARN o.j.d.l.c.ConsistentKeyLocker RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/grouptypes EntryTimestamp=2021-07-13T00:12:19.867Z InvocationID=a9635568-78a8-46b6-b86a-a81c6c592e9f PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Lock write succeeded but took too long: duration PT8.155326S exceeded limit PT0.5S 2021-07-13T00:13:07.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:13:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:13:07.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:13:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:13:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:13:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:13:10.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:13:13.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:13:14.621Z [JanusGraph-Health-Check-Thread] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.execute(VertexCentricQueryBuilder.java:73) at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.properties(VertexCentricQueryBuilder.java:100) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Vertex.property(Vertex.java:70) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:341) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:336) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 0 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ... 4 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 0 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 0 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:13:16.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:13:16.922Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b02e1966-5475-4c1b-801a-f3bb45a18826 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:13:16.922Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b02e1966-5475-4c1b-801a-f3bb45a18826 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64595dee 2021-07-13T00:13:16.922Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b02e1966-5475-4c1b-801a-f3bb45a18826 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:13:16.922Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b02e1966-5475-4c1b-801a-f3bb45a18826 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:13:17.466Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b02e1966-5475-4c1b-801a-f3bb45a18826 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:13:17.466Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=b02e1966-5475-4c1b-801a-f3bb45a18826 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:13:18.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a51ab995-d056-4296-8bba-ef4dbb9b82bd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:13:18.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a51ab995-d056-4296-8bba-ef4dbb9b82bd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5fe0fa15 2021-07-13T00:13:18.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a51ab995-d056-4296-8bba-ef4dbb9b82bd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:13:18.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a51ab995-d056-4296-8bba-ef4dbb9b82bd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:13:18.667Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:13:20.619Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02c3b031-a84a-4f40-be32-b78450ee9904 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:13:20.619Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02c3b031-a84a-4f40-be32-b78450ee9904 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2b79a741 2021-07-13T00:13:20.619Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02c3b031-a84a-4f40-be32-b78450ee9904 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:13:20.619Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02c3b031-a84a-4f40-be32-b78450ee9904 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:13:20.962Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=02c3b031-a84a-4f40-be32-b78450ee9904 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:13:20.963Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=02c3b031-a84a-4f40-be32-b78450ee9904 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:13:22.079Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94aaafef-71f8-450c-8556-c6618282530b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:13:22.079Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94aaafef-71f8-450c-8556-c6618282530b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e129ed 2021-07-13T00:13:22.079Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94aaafef-71f8-450c-8556-c6618282530b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:13:22.079Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94aaafef-71f8-450c-8556-c6618282530b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:13:22.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:13:22.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:13:23.488Z [qtp817348612-101] WARN o.o.s.be.servlets.TypesUploadServlet RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/grouptypes EntryTimestamp=2021-07-13T00:12:19.867Z InvocationID=a9635568-78a8-46b6-b86a-a81c6c592e9f PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp #uploadGroupTypes('{"tosca.groups.Root": {"displayName": "Root", "icon": "group-root"}, "org.openecomp.groups.heat.HeatStack": {"displayName": "Heat Stack", "icon": "group-heat-stack"}, "org.openecomp.groups.VfModule": {"displayName": "VF Module", "icon": "group-vf-module"}, "org.openecomp.groups.NetworkCollection": {"displayName": "Network Collection", "icon": "group-network-collection"}, "org.openecomp.groups.VfcInstanceGroup": {"displayName": "Vfc Instance", "icon": "group-vfc-instance"}, "tosca.groups.nfv.PlacementGroup": {"displayName": "NFV PlacementGroup", "icon": "group-nfv-placementgroup"}, "tosca.groups.nfv.NsPlacementGroup": {"displayName": "NFV NS PlacementGroup", "icon": "group-nfv-placementgroup"}, "tosca.groups.nfv.VNFFG": {"displayName": "NFV VNF Forwarding Graph", "icon": "group-nfv-placementgroup"}}', /tmp/rep17375620736714331474tmp, 'Request(POST https://sdc-be.onap:8443/sdc2/rest/v1/catalog/uploadType/grouptypes)@62228854', 'jh0003'): org.apache.cxf.jaxrs.impl.ResponseImpl@5e205e2a in 1.06min (too slow!) 2021-07-13T00:13:23.489Z [qtp817348612-101] INFO o.o.l.f.base.AbstractAuditLogFilter ResponseCode=201 RequestID=null ElapsedTime=63621 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp StatusCode=COMPLETE ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/grouptypes EntryTimestamp=2021-07-13T00:12:19.867Z InvocationID=a9635568-78a8-46b6-b86a-a81c6c592e9f LogTimestamp=2021-07-13T00:13:23.488808Z ResponseDesc=Created Exiting. 2021-07-13T00:13:23.490Z [qtp817348612-101] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc2/rest/v1/catalog/uploadType/grouptypes HTTP/1.1 SC="201" 2021-07-13T00:13:23.565Z [qtp817348612-16] INFO o.o.l.f.base.AbstractAuditLogFilter RequestID=0b452617-5374-4763-9ba5-aab34e50b2e6 InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS RemoteHost=10.242.77.79 ServiceName=v1/catalog/uploadType/policytypes ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:13:23.565739Z InvocationID=119e1f3d-56a1-4a85-867e-c545f6a6884f LogTimestamp=2021-07-13T00:13:23.565783Z Entering 2021-07-13T00:13:23.565Z [qtp817348612-16] INFO o.o.sdc.be.filters.BeServletFilter RequestID=0b452617-5374-4763-9ba5-aab34e50b2e6 InstanceID=d27371ad-79e3-41f9-bc44-ff554808ee1e ElapsedTime=0 Severity=0 PartnerName=testName ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp ClientIPAddress=10.242.77.79 StatusCode=INPROGRESS userId=jh0003 RemoteHost=10.242.77.79 ServiceName=v1/catalog/uploadType/policytypes ServerIPAddress=10.242.98.212 ServiceInstanceID=null EntryTimestamp=2021-07-13T00:13:23.565739Z InvocationID=119e1f3d-56a1-4a85-867e-c545f6a6884f LogTimestamp=2021-07-13T00:13:23.565783Z localAddr=10.242.98.212 remoteAddr=10.242.77.79 POST /sdc2/rest/v1/catalog/uploadType/policytypes HTTP/1.1 2021-07-13T00:13:23.607Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.Root to dao for create 2021-07-13T00:13:23.865Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.Placement to dao for create 2021-07-13T00:13:24.061Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.Scaling to dao for create 2021-07-13T00:13:24.340Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:13:25.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:13:25.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:13:25.528Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.Update to dao for create 2021-07-13T00:13:25.568Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.Performance to dao for create 2021-07-13T00:13:25.665Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : org.openecomp.policies.placement.Antilocate to dao for create 2021-07-13T00:13:25.967Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : org.openecomp.policies.placement.Colocate to dao for create 2021-07-13T00:13:26.323Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : org.openecomp.policies.placement.valet.Diversity to dao for create 2021-07-13T00:13:26.372Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : org.openecomp.policies.placement.valet.Exclusivity to dao for create 2021-07-13T00:13:26.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e78ce8b-07e2-442c-a22c-aa57d5bd1196 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:13:26.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e78ce8b-07e2-442c-a22c-aa57d5bd1196 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f9a83b 2021-07-13T00:13:26.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e78ce8b-07e2-442c-a22c-aa57d5bd1196 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:13:26.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e78ce8b-07e2-442c-a22c-aa57d5bd1196 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:13:26.728Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : org.openecomp.policies.placement.valet.Affinity to dao for create 2021-07-13T00:13:27.066Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : org.openecomp.policies.scaling.Fixed to dao for create 2021-07-13T00:13:27.166Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : org.openecomp.policies.External to dao for create 2021-07-13T00:13:27.522Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.nfv.InstantiationLevels to dao for create 2021-07-13T00:13:27.561Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:13:28.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=939d88e2-c0a8-4af3-a3ef-59d5422358a0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:13:28.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=939d88e2-c0a8-4af3-a3ef-59d5422358a0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f11da83 2021-07-13T00:13:28.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=939d88e2-c0a8-4af3-a3ef-59d5422358a0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:13:28.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=939d88e2-c0a8-4af3-a3ef-59d5422358a0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:13:28.816Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=939d88e2-c0a8-4af3-a3ef-59d5422358a0 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:13:28.816Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=939d88e2-c0a8-4af3-a3ef-59d5422358a0 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:13:30.012Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=014c9edd-0bb1-4c10-9b53-74b06bea1be9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:13:30.013Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=014c9edd-0bb1-4c10-9b53-74b06bea1be9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@39343d8f 2021-07-13T00:13:30.013Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=014c9edd-0bb1-4c10-9b53-74b06bea1be9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:13:30.013Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=014c9edd-0bb1-4c10-9b53-74b06bea1be9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:13:30.462Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=014c9edd-0bb1-4c10-9b53-74b06bea1be9 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:13:30.462Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=014c9edd-0bb1-4c10-9b53-74b06bea1be9 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:13:30.665Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.nfv.VduInstantiationLevels to dao for create 2021-07-13T00:13:31.614Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=882e061b-ebf0-42cc-862e-a242eb430f89 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:13:31.614Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=882e061b-ebf0-42cc-862e-a242eb430f89 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3516d505 2021-07-13T00:13:31.614Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=882e061b-ebf0-42cc-862e-a242eb430f89 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:13:31.614Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=882e061b-ebf0-42cc-862e-a242eb430f89 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:13:32.763Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.nfv.VirtualLinkInstantiationLevels to dao for create 2021-07-13T00:13:32.867Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.nfv.ScalingAspects to dao for create 2021-07-13T00:13:33.463Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.nfv.VduScalingAspectDeltas to dao for create 2021-07-13T00:13:33.860Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.nfv.VirtualLinkBitrateScalingAspectDeltas to dao for create 2021-07-13T00:13:34.121Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.nfv.VduInitialDelta to dao for create 2021-07-13T00:13:34.460Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.nfv.VirtualLinkBitrateInitialDelta to dao for create 2021-07-13T00:13:35.126Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3ebf321-fba8-45aa-88ba-baa55024a830 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:13:35.126Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3ebf321-fba8-45aa-88ba-baa55024a830 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7fe40bec 2021-07-13T00:13:35.126Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3ebf321-fba8-45aa-88ba-baa55024a830 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:13:35.126Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3ebf321-fba8-45aa-88ba-baa55024a830 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:13:36.471Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d50c256a-99d2-465e-a7b8-e9b2ef1164aa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:13:36.471Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d50c256a-99d2-465e-a7b8-e9b2ef1164aa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ef01351 2021-07-13T00:13:36.471Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d50c256a-99d2-465e-a7b8-e9b2ef1164aa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:13:36.471Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d50c256a-99d2-465e-a7b8-e9b2ef1164aa PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:13:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:13:37.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:13:38.430Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03d764a8-e2b0-47b7-9489-241d087ef6b4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:13:38.430Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03d764a8-e2b0-47b7-9489-241d087ef6b4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@36b08b45 2021-07-13T00:13:38.430Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03d764a8-e2b0-47b7-9489-241d087ef6b4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:13:38.430Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03d764a8-e2b0-47b7-9489-241d087ef6b4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:13:38.917Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.nfv.SecurityGroupRule to dao for create 2021-07-13T00:13:39.765Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.nfv.SupportedVnfInterface to dao for create 2021-07-13T00:13:40.314Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf8c24af-c3c2-433a-b997-3c16d5d38bad PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:13:40.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf8c24af-c3c2-433a-b997-3c16d5d38bad PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@350dde6b 2021-07-13T00:13:40.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf8c24af-c3c2-433a-b997-3c16d5d38bad PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:13:40.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf8c24af-c3c2-433a-b997-3c16d5d38bad PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:13:49.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:13:49.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:13:51.722Z [cluster1-nio-worker-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.199.132:9042 com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:215) at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:242) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:13:51.722Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.199.132:9042 com.datastax.driver.core.exceptions.ConnectionException: [cassandra.onap/10.242.199.132:9042] Pool was closed during initialization at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:173) at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:158) at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1080) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.google.common.util.concurrent.CollectionFuture.handleAllCompleted(CollectionFuture.java:69) at com.google.common.util.concurrent.AggregateFuture.processCompleted(AggregateFuture.java:282) at com.google.common.util.concurrent.AggregateFuture.decrementCountAndMaybeComplete(AggregateFuture.java:264) at com.google.common.util.concurrent.AggregateFuture.access$200(AggregateFuture.java:42) at com.google.common.util.concurrent.AggregateFuture$1.run(AggregateFuture.java:147) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:100) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:215) at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:242) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:13:51.723Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0884d099-aaf6-4ce4-abee-72a2dea01b2f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:13:51.723Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0884d099-aaf6-4ce4-abee-72a2dea01b2f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6a69184d 2021-07-13T00:13:51.723Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0884d099-aaf6-4ce4-abee-72a2dea01b2f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:13:51.723Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0884d099-aaf6-4ce4-abee-72a2dea01b2f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:13:52.524Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:13:52.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:13:52.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:13:54.729Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=0884d099-aaf6-4ce4-abee-72a2dea01b2f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:13:54.729Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=0884d099-aaf6-4ce4-abee-72a2dea01b2f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:13:54.729Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=0884d099-aaf6-4ce4-abee-72a2dea01b2f ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:13:54.729Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=0884d099-aaf6-4ce4-abee-72a2dea01b2f ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:13:54.859Z [qtp817348612-16] WARN o.j.d.l.c.ConsistentKeyLocker RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Lock write succeeded but took too long: duration PT12.088211S exceeded limit PT0.5S 2021-07-13T00:13:55.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b7c0cad-e60b-4b4c-9ac8-f722e19a8664 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:13:55.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b7c0cad-e60b-4b4c-9ac8-f722e19a8664 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@44ade4ca 2021-07-13T00:13:55.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b7c0cad-e60b-4b4c-9ac8-f722e19a8664 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:13:55.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b7c0cad-e60b-4b4c-9ac8-f722e19a8664 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:13:58.827Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=4b7c0cad-e60b-4b4c-9ac8-f722e19a8664 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:13:58.827Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=4b7c0cad-e60b-4b4c-9ac8-f722e19a8664 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:13:59.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e975813-41d7-4c52-a7a8-cb8dd22969eb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:13:59.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e975813-41d7-4c52-a7a8-cb8dd22969eb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d50d39a 2021-07-13T00:13:59.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e975813-41d7-4c52-a7a8-cb8dd22969eb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:13:59.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e975813-41d7-4c52-a7a8-cb8dd22969eb PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:14:01.568Z [qtp817348612-16] WARN o.j.d.l.c.ConsistentKeyLocker RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Lock write succeeded but took too long: duration PT5.708449S exceeded limit PT0.5S 2021-07-13T00:14:02.915Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=2e975813-41d7-4c52-a7a8-cb8dd22969eb ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:14:02.915Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=2e975813-41d7-4c52-a7a8-cb8dd22969eb ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:14:04.215Z [qtp817348612-16] WARN o.j.d.l.c.ConsistentKeyLocker RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Lock write succeeded but took too long: duration PT2.249087S exceeded limit PT0.5S 2021-07-13T00:14:04.815Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a2dfa45-f849-47a2-8eb8-ad480e96420d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:14:04.815Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a2dfa45-f849-47a2-8eb8-ad480e96420d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@24f93388 2021-07-13T00:14:04.815Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a2dfa45-f849-47a2-8eb8-ad480e96420d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:14:04.815Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a2dfa45-f849-47a2-8eb8-ad480e96420d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:14:05.264Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=1a2dfa45-f849-47a2-8eb8-ad480e96420d ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:14:05.264Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=1a2dfa45-f849-47a2-8eb8-ad480e96420d ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:14:06.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d12cdf52-0e07-47bf-a3c8-8f306a9a9920 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:14:06.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d12cdf52-0e07-47bf-a3c8-8f306a9a9920 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@541d52a5 2021-07-13T00:14:06.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d12cdf52-0e07-47bf-a3c8-8f306a9a9920 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:14:06.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d12cdf52-0e07-47bf-a3c8-8f306a9a9920 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:14:07.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:14:07.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:14:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:14:09.369Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:14:09.369Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:14:10.731Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c86809c-34b0-4142-a3a1-e3cd5500d316 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:14:10.732Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c86809c-34b0-4142-a3a1-e3cd5500d316 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@136b8c4e 2021-07-13T00:14:10.732Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c86809c-34b0-4142-a3a1-e3cd5500d316 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:14:10.732Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c86809c-34b0-4142-a3a1-e3cd5500d316 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:14:13.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:14:13.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:14:14.616Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2afb411-379e-409e-9a6f-ba082a4bab4f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:14:14.616Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2afb411-379e-409e-9a6f-ba082a4bab4f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7e6388d8 2021-07-13T00:14:14.616Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2afb411-379e-409e-9a6f-ba082a4bab4f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:14:14.616Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2afb411-379e-409e-9a6f-ba082a4bab4f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:14:15.468Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:14:16.627Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=951065df-0120-4311-b807-ae610b7a2747 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:14:16.627Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=951065df-0120-4311-b807-ae610b7a2747 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5524e1dd 2021-07-13T00:14:16.627Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=951065df-0120-4311-b807-ae610b7a2747 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:14:16.627Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=951065df-0120-4311-b807-ae610b7a2747 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:14:16.946Z [qtp817348612-16] INFO o.j.d.util.BackendOperation RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Temporary exception during backend operation [CacheMutation]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateManyUnlogged(CQLStoreManager.java:504) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateMany(CQLStoreManager.java:435) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStoreManager.mutateMany(ExpectedValueCheckingStoreManager.java:76) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:94) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:91) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:91) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:139) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:196) at org.janusgraph.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:134) at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:733) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1379) at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.doCommit(JanusGraphBlueprintsGraph.java:298) at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:104) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.commit(JanusGraphClient.java:250) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.commit(JanusGraphGenericDao.java:74) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.handleTransactionCommitRollback(JanusGraphGenericDao.java:87) at org.openecomp.sdc.be.model.operations.impl.GroupTypeOperation.getGroupTypeByCriteria(GroupTypeOperation.java:346) at org.openecomp.sdc.be.model.operations.impl.GroupTypeOperation.getLatestGroupTypeByType(GroupTypeOperation.java:322) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.validatePolicyType(PolicyTypeImportManager.java:103) at org.openecomp.sdc.be.components.impl.CommonImportManager.handleType(CommonImportManager.java:305) at org.openecomp.sdc.be.components.impl.CommonImportManager.createElementTypesByDao(CommonImportManager.java:268) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.upsertPolicyTypesByDao(PolicyTypeImportManager.java:74) at fj.data.Either$LeftProjection.bind(Either.java:287) at org.openecomp.sdc.be.components.impl.CommonImportManager.createElementTypes(CommonImportManager.java:392) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.createPolicyTypes(PolicyTypeImportManager.java:65) at org.openecomp.sdc.be.servlets.TypesUploadServlet.lambda$createPolicyTypes$8(TypesUploadServlet.java:334) at org.openecomp.sdc.be.servlets.TypesUploadServlet.buildStatusForElementTypeCreate(TypesUploadServlet.java:343) at org.openecomp.sdc.be.servlets.TypesUploadServlet.createPolicyTypes(TypesUploadServlet.java:335) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadTypesWithMetaData(TypesUploadServlet.java:287) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadPolicyTypes_aroundBody12(TypesUploadServlet.java:228) at org.openecomp.sdc.be.servlets.TypesUploadServlet$AjcClosure13.run(TypesUploadServlet.java:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:212) at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) at com.jcabi.aspects.aj.MethodLogger.wrapClass(MethodLogger.java:136) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadPolicyTypes(TypesUploadServlet.java:227) at org.openecomp.sdc.be.servlets.TypesUploadServlet$$FastClassBySpringCGLIB$$893260aa.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at org.openecomp.sdc.be.servlets.TypesUploadServlet$$EnhancerBySpringCGLIB$$8228831e.uploadPolicyTypes() at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1411) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:226) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.togglz.servlet.TogglzFilter.doFilter(TogglzFilter.java:100) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.ReqValidationFilter.doFilter(ReqValidationFilter.java:72) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.GatewayFilter.doFilter(GatewayFilter.java:71) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1630) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:501) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:113) at com.datastax.driver.core.Responses$Error.asException(Responses.java:167) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:87) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:14:16.968Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=951065df-0120-4311-b807-ae610b7a2747 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:14:16.968Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=951065df-0120-4311-b807-ae610b7a2747 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:14:19.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:14:19.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:14:22.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4dd8be47-3258-4b08-8c0d-d001c29eff6a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:14:22.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4dd8be47-3258-4b08-8c0d-d001c29eff6a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6b1209a6 2021-07-13T00:14:22.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4dd8be47-3258-4b08-8c0d-d001c29eff6a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:14:22.114Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4dd8be47-3258-4b08-8c0d-d001c29eff6a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:14:22.266Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:14:22.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:14:22.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:14:25.117Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=4dd8be47-3258-4b08-8c0d-d001c29eff6a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:14:25.117Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=4dd8be47-3258-4b08-8c0d-d001c29eff6a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:14:25.118Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=4dd8be47-3258-4b08-8c0d-d001c29eff6a ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:14:25.118Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=4dd8be47-3258-4b08-8c0d-d001c29eff6a ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:14:25.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:14:25.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:14:28.461Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:14:30.023Z [qtp817348612-16] WARN o.j.d.l.c.ConsistentKeyLocker RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Temporary storage exception while deleting lock org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateManyUnlogged(CQLStoreManager.java:504) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateMany(CQLStoreManager.java:435) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.mutate(CQLKeyColumnValueStore.java:363) at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.deleteSingleLock(ConsistentKeyLocker.java:538) at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.deleteSingleLock(ConsistentKeyLocker.java:118) at org.janusgraph.diskstorage.locking.AbstractLocker.deleteLocks(AbstractLocker.java:381) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStore.deleteLocks(ExpectedValueCheckingStore.java:115) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.deleteAllLocks(ExpectedValueCheckingTransaction.java:282) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.commit(ExpectedValueCheckingTransaction.java:92) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:197) at org.janusgraph.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:134) at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:733) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1379) at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.doCommit(JanusGraphBlueprintsGraph.java:298) at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:104) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.commit(JanusGraphClient.java:250) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.commit(JanusGraphGenericDao.java:74) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.handleTransactionCommitRollback(JanusGraphGenericDao.java:87) at org.openecomp.sdc.be.model.operations.impl.GroupTypeOperation.getGroupTypeByCriteria(GroupTypeOperation.java:346) at org.openecomp.sdc.be.model.operations.impl.GroupTypeOperation.getLatestGroupTypeByType(GroupTypeOperation.java:322) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.validatePolicyType(PolicyTypeImportManager.java:103) at org.openecomp.sdc.be.components.impl.CommonImportManager.handleType(CommonImportManager.java:305) at org.openecomp.sdc.be.components.impl.CommonImportManager.createElementTypesByDao(CommonImportManager.java:268) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.upsertPolicyTypesByDao(PolicyTypeImportManager.java:74) at fj.data.Either$LeftProjection.bind(Either.java:287) at org.openecomp.sdc.be.components.impl.CommonImportManager.createElementTypes(CommonImportManager.java:392) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.createPolicyTypes(PolicyTypeImportManager.java:65) at org.openecomp.sdc.be.servlets.TypesUploadServlet.lambda$createPolicyTypes$8(TypesUploadServlet.java:334) at org.openecomp.sdc.be.servlets.TypesUploadServlet.buildStatusForElementTypeCreate(TypesUploadServlet.java:343) at org.openecomp.sdc.be.servlets.TypesUploadServlet.createPolicyTypes(TypesUploadServlet.java:335) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadTypesWithMetaData(TypesUploadServlet.java:287) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadPolicyTypes_aroundBody12(TypesUploadServlet.java:228) at org.openecomp.sdc.be.servlets.TypesUploadServlet$AjcClosure13.run(TypesUploadServlet.java:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:212) at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) at com.jcabi.aspects.aj.MethodLogger.wrapClass(MethodLogger.java:136) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadPolicyTypes(TypesUploadServlet.java:227) at org.openecomp.sdc.be.servlets.TypesUploadServlet$$FastClassBySpringCGLIB$$893260aa.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at org.openecomp.sdc.be.servlets.TypesUploadServlet$$EnhancerBySpringCGLIB$$8228831e.uploadPolicyTypes() at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1411) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:226) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.togglz.servlet.TogglzFilter.doFilter(TogglzFilter.java:100) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.ReqValidationFilter.doFilter(ReqValidationFilter.java:72) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.GatewayFilter.doFilter(GatewayFilter.java:71) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1630) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:501) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 0 acknowledged the write) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 0 acknowledged the write) at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:113) at com.datastax.driver.core.Responses$Error.asException(Responses.java:167) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 0 acknowledged the write) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:87) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:14:30.023Z [ApplicationDataTypeCacheThread-1] INFO o.j.d.util.BackendOperation RequestID=null ErrorCategory=INFO ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=500 TargetEntity=ReplaceDataTypesCache Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Element.keys(Element.java:69) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getProperties(JanusGraphGenericDao.java:1491) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getByCriteria(JanusGraphGenericDao.java:759) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAllDataTypeNodes(PropertyOperation.java:1566) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.findAllDataTypesLazy(ApplicationDataTypeCache.java:309) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.hasDataTypesChanged(ApplicationDataTypeCache.java:220) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.run(ApplicationDataTypeCache.java:200) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:308) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:422) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 26 common frames omitted 2021-07-13T00:14:31.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:14:31.910Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:14:34.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:14:37.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:14:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:14:37.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:14:40.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:14:43.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:14:45.166Z [cluster1-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.199.132:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.199.132:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:14:45.167Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4da66b66-ea1f-4eba-b7da-6c822f261b58 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:14:45.167Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4da66b66-ea1f-4eba-b7da-6c822f261b58 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@24d4f176 2021-07-13T00:14:45.167Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4da66b66-ea1f-4eba-b7da-6c822f261b58 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:14:45.167Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4da66b66-ea1f-4eba-b7da-6c822f261b58 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:14:46.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:14:48.243Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=4da66b66-ea1f-4eba-b7da-6c822f261b58 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:14:48.243Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=4da66b66-ea1f-4eba-b7da-6c822f261b58 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:14:49.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d1d256e-7738-490c-807c-8e2ff5793539 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:14:49.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d1d256e-7738-490c-807c-8e2ff5793539 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@34cba36a 2021-07-13T00:14:49.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d1d256e-7738-490c-807c-8e2ff5793539 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:14:49.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d1d256e-7738-490c-807c-8e2ff5793539 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:14:49.461Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:14:52.310Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=6d1d256e-7738-490c-807c-8e2ff5793539 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:14:52.311Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=6d1d256e-7738-490c-807c-8e2ff5793539 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:14:52.312Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6d1d256e-7738-490c-807c-8e2ff5793539 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:14:52.312Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=6d1d256e-7738-490c-807c-8e2ff5793539 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:14:52.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:14:52.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:14:52.931Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:14:53.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a758970e-558d-4a83-b340-fbe3bae6275a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:14:53.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a758970e-558d-4a83-b340-fbe3bae6275a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4c33c4b6 2021-07-13T00:14:53.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a758970e-558d-4a83-b340-fbe3bae6275a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:14:53.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a758970e-558d-4a83-b340-fbe3bae6275a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:14:55.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:14:56.208Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=a758970e-558d-4a83-b340-fbe3bae6275a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:14:56.208Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=a758970e-558d-4a83-b340-fbe3bae6275a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:14:57.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b09d572-d48e-4fae-a56b-1dea9ce10571 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:14:57.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b09d572-d48e-4fae-a56b-1dea9ce10571 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f80c46a 2021-07-13T00:14:57.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b09d572-d48e-4fae-a56b-1dea9ce10571 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:14:57.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b09d572-d48e-4fae-a56b-1dea9ce10571 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:14:58.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:15:00.409Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=9b09d572-d48e-4fae-a56b-1dea9ce10571 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:15:00.410Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=9b09d572-d48e-4fae-a56b-1dea9ce10571 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:15:01.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4b46dd1-f9ab-48f8-a162-0db82b515e9d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:15:01.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4b46dd1-f9ab-48f8-a162-0db82b515e9d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c81e74f 2021-07-13T00:15:01.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4b46dd1-f9ab-48f8-a162-0db82b515e9d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:15:01.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4b46dd1-f9ab-48f8-a162-0db82b515e9d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:15:01.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:15:04.113Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=a4b46dd1-f9ab-48f8-a162-0db82b515e9d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:15:04.113Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=a4b46dd1-f9ab-48f8-a162-0db82b515e9d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:15:04.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:15:05.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79296107-eb7e-40f7-b684-ae9fd27a0cad PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:15:05.367Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79296107-eb7e-40f7-b684-ae9fd27a0cad PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3c121e69 2021-07-13T00:15:05.367Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79296107-eb7e-40f7-b684-ae9fd27a0cad PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:15:05.367Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79296107-eb7e-40f7-b684-ae9fd27a0cad PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:15:07.460Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:15:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:15:07.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:15:08.426Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=79296107-eb7e-40f7-b684-ae9fd27a0cad ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:15:08.426Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=79296107-eb7e-40f7-b684-ae9fd27a0cad ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:15:09.228Z [cluster1-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-07-13T00:15:09.229Z [cluster1-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.199.132:9042, scheduling retry in 2000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:15:09.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7fb70612-cdd0-4182-827a-312de3a5e468 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:15:09.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7fb70612-cdd0-4182-827a-312de3a5e468 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4d7fb510 2021-07-13T00:15:09.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7fb70612-cdd0-4182-827a-312de3a5e468 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:15:09.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7fb70612-cdd0-4182-827a-312de3a5e468 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:15:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:15:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:15:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:15:10.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:15:12.408Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=7fb70612-cdd0-4182-827a-312de3a5e468 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:15:12.409Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=7fb70612-cdd0-4182-827a-312de3a5e468 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:15:13.361Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=58478c10-90e2-444d-a080-8e1756c9d8c0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:15:13.361Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=58478c10-90e2-444d-a080-8e1756c9d8c0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@657712d8 2021-07-13T00:15:13.361Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=58478c10-90e2-444d-a080-8e1756c9d8c0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:15:13.362Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=58478c10-90e2-444d-a080-8e1756c9d8c0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:15:13.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:15:14.138Z [qtp817348612-16] WARN o.j.d.l.c.ConsistentKeyLocker RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Temporary storage exception while deleting lock org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateManyUnlogged(CQLStoreManager.java:504) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateMany(CQLStoreManager.java:435) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.mutate(CQLKeyColumnValueStore.java:363) at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.deleteSingleLock(ConsistentKeyLocker.java:538) at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.deleteSingleLock(ConsistentKeyLocker.java:118) at org.janusgraph.diskstorage.locking.AbstractLocker.deleteLocks(AbstractLocker.java:381) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStore.deleteLocks(ExpectedValueCheckingStore.java:115) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.deleteAllLocks(ExpectedValueCheckingTransaction.java:282) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.commit(ExpectedValueCheckingTransaction.java:92) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:197) at org.janusgraph.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:134) at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:733) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1379) at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.doCommit(JanusGraphBlueprintsGraph.java:298) at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:104) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.commit(JanusGraphClient.java:250) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.commit(JanusGraphGenericDao.java:74) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.handleTransactionCommitRollback(JanusGraphGenericDao.java:87) at org.openecomp.sdc.be.model.operations.impl.GroupTypeOperation.getGroupTypeByCriteria(GroupTypeOperation.java:346) at org.openecomp.sdc.be.model.operations.impl.GroupTypeOperation.getLatestGroupTypeByType(GroupTypeOperation.java:322) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.validatePolicyType(PolicyTypeImportManager.java:103) at org.openecomp.sdc.be.components.impl.CommonImportManager.handleType(CommonImportManager.java:305) at org.openecomp.sdc.be.components.impl.CommonImportManager.createElementTypesByDao(CommonImportManager.java:268) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.upsertPolicyTypesByDao(PolicyTypeImportManager.java:74) at fj.data.Either$LeftProjection.bind(Either.java:287) at org.openecomp.sdc.be.components.impl.CommonImportManager.createElementTypes(CommonImportManager.java:392) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.createPolicyTypes(PolicyTypeImportManager.java:65) at org.openecomp.sdc.be.servlets.TypesUploadServlet.lambda$createPolicyTypes$8(TypesUploadServlet.java:334) at org.openecomp.sdc.be.servlets.TypesUploadServlet.buildStatusForElementTypeCreate(TypesUploadServlet.java:343) at org.openecomp.sdc.be.servlets.TypesUploadServlet.createPolicyTypes(TypesUploadServlet.java:335) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadTypesWithMetaData(TypesUploadServlet.java:287) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadPolicyTypes_aroundBody12(TypesUploadServlet.java:228) at org.openecomp.sdc.be.servlets.TypesUploadServlet$AjcClosure13.run(TypesUploadServlet.java:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:212) at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) at com.jcabi.aspects.aj.MethodLogger.wrapClass(MethodLogger.java:136) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadPolicyTypes(TypesUploadServlet.java:227) at org.openecomp.sdc.be.servlets.TypesUploadServlet$$FastClassBySpringCGLIB$$893260aa.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at org.openecomp.sdc.be.servlets.TypesUploadServlet$$EnhancerBySpringCGLIB$$8228831e.uploadPolicyTypes() at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1411) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:226) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.togglz.servlet.TogglzFilter.doFilter(TogglzFilter.java:100) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.ReqValidationFilter.doFilter(ReqValidationFilter.java:72) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.GatewayFilter.doFilter(GatewayFilter.java:71) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1630) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:501) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 0 acknowledged the write) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 0 acknowledged the write) at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:113) at com.datastax.driver.core.Responses$Error.asException(Responses.java:167) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 0 acknowledged the write) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:87) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:15:14.618Z [pool-8-thread-1] WARN o.j.diskstorage.log.kcvs.KCVSLog Could not read messages for timestamp [2021-07-13T00:15:00Z] (this read will be retried) org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:56) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:726) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:100) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) ... 8 common frames omitted Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:792) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:789) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) ... 9 common frames omitted Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:15:14.630Z [ApplicationDataTypeCacheThread-1] INFO o.j.d.util.BackendOperation RequestID=null ErrorCategory=INFO ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=500 TargetEntity=ReplaceDataTypesCache Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Element.keys(Element.java:69) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getProperties(JanusGraphGenericDao.java:1491) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getByCriteria(JanusGraphGenericDao.java:759) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAllDataTypeNodes(PropertyOperation.java:1566) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.findAllDataTypesLazy(ApplicationDataTypeCache.java:309) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.hasDataTypesChanged(ApplicationDataTypeCache.java:220) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.run(ApplicationDataTypeCache.java:200) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:15:16.416Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=58478c10-90e2-444d-a080-8e1756c9d8c0 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:15:16.416Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=58478c10-90e2-444d-a080-8e1756c9d8c0 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:15:16.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:15:17.463Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f61528d-f39a-418a-9943-3a4051fd0d54 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:15:17.463Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f61528d-f39a-418a-9943-3a4051fd0d54 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@37752161 2021-07-13T00:15:17.463Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f61528d-f39a-418a-9943-3a4051fd0d54 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:15:17.463Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f61528d-f39a-418a-9943-3a4051fd0d54 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:15:19.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:15:20.215Z [JanusGraph-Health-Check-Thread] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [VertexIndexQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:399) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:396) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.indexQuery(BackendTransaction.java:396) at org.janusgraph.graphdb.query.graph.MultiKeySliceQuery.execute(MultiKeySliceQuery.java:51) at org.janusgraph.graphdb.database.IndexSerializer.query(IndexSerializer.java:512) at org.janusgraph.graphdb.util.SubqueryIterator.(SubqueryIterator.java:66) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1295) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1190) at org.janusgraph.graphdb.query.QueryProcessor$LimitAdjustingIterator.getNewIterator(QueryProcessor.java:194) at org.janusgraph.graphdb.query.LimitAdjustingIterator.hasNext(LimitAdjustingIterator.java:68) at org.janusgraph.graphdb.query.ResultSetIterator.nextInternal(ResultSetIterator.java:54) at org.janusgraph.graphdb.query.ResultSetIterator.(ResultSetIterator.java:44) at org.janusgraph.graphdb.query.QueryProcessor.iterator(QueryProcessor.java:66) at com.google.common.collect.Iterables$4.iterator(Iterables.java:559) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:340) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:336) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ... 4 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:308) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:422) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 26 common frames omitted 2021-07-13T00:15:20.225Z [qtp817348612-16] WARN o.j.d.l.c.ConsistentKeyLocker RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Temporary storage exception while deleting lock org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateManyUnlogged(CQLStoreManager.java:504) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateMany(CQLStoreManager.java:435) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.mutate(CQLKeyColumnValueStore.java:363) at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.deleteSingleLock(ConsistentKeyLocker.java:538) at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.deleteSingleLock(ConsistentKeyLocker.java:118) at org.janusgraph.diskstorage.locking.AbstractLocker.deleteLocks(AbstractLocker.java:381) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStore.deleteLocks(ExpectedValueCheckingStore.java:115) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.deleteAllLocks(ExpectedValueCheckingTransaction.java:282) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.commit(ExpectedValueCheckingTransaction.java:92) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:197) at org.janusgraph.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:134) at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:733) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1379) at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.doCommit(JanusGraphBlueprintsGraph.java:298) at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:104) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.commit(JanusGraphClient.java:250) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.commit(JanusGraphGenericDao.java:74) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.handleTransactionCommitRollback(JanusGraphGenericDao.java:87) at org.openecomp.sdc.be.model.operations.impl.GroupTypeOperation.getGroupTypeByCriteria(GroupTypeOperation.java:346) at org.openecomp.sdc.be.model.operations.impl.GroupTypeOperation.getLatestGroupTypeByType(GroupTypeOperation.java:322) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.validatePolicyType(PolicyTypeImportManager.java:103) at org.openecomp.sdc.be.components.impl.CommonImportManager.handleType(CommonImportManager.java:305) at org.openecomp.sdc.be.components.impl.CommonImportManager.createElementTypesByDao(CommonImportManager.java:268) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.upsertPolicyTypesByDao(PolicyTypeImportManager.java:74) at fj.data.Either$LeftProjection.bind(Either.java:287) at org.openecomp.sdc.be.components.impl.CommonImportManager.createElementTypes(CommonImportManager.java:392) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.createPolicyTypes(PolicyTypeImportManager.java:65) at org.openecomp.sdc.be.servlets.TypesUploadServlet.lambda$createPolicyTypes$8(TypesUploadServlet.java:334) at org.openecomp.sdc.be.servlets.TypesUploadServlet.buildStatusForElementTypeCreate(TypesUploadServlet.java:343) at org.openecomp.sdc.be.servlets.TypesUploadServlet.createPolicyTypes(TypesUploadServlet.java:335) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadTypesWithMetaData(TypesUploadServlet.java:287) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadPolicyTypes_aroundBody12(TypesUploadServlet.java:228) at org.openecomp.sdc.be.servlets.TypesUploadServlet$AjcClosure13.run(TypesUploadServlet.java:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:212) at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) at com.jcabi.aspects.aj.MethodLogger.wrapClass(MethodLogger.java:136) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadPolicyTypes(TypesUploadServlet.java:227) at org.openecomp.sdc.be.servlets.TypesUploadServlet$$FastClassBySpringCGLIB$$893260aa.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at org.openecomp.sdc.be.servlets.TypesUploadServlet$$EnhancerBySpringCGLIB$$8228831e.uploadPolicyTypes() at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1411) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:226) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.togglz.servlet.TogglzFilter.doFilter(TogglzFilter.java:100) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.ReqValidationFilter.doFilter(ReqValidationFilter.java:72) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.GatewayFilter.doFilter(GatewayFilter.java:71) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1630) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:501) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:113) at com.datastax.driver.core.Responses$Error.asException(Responses.java:167) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:308) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:422) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:87) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 26 common frames omitted 2021-07-13T00:15:20.225Z [ApplicationDataTypeCacheThread-1] ERROR o.o.s.b.m.c.ApplicationDataTypeCache RequestID=null ErrorCategory=ERROR ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=900 TargetEntity=null Failed to fetch data types from database. Status is GENERAL_ERROR 2021-07-13T00:15:20.225Z [ApplicationDataTypeCacheThread-1] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=null ErrorCategory=ERROR ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=200 TargetEntity=ApplicationDataTypesCache Internal Connection problem. Description: Failed to fetch data types from database. Status is GENERAL_ERROR 2021-07-13T00:15:20.225Z [ApplicationDataTypeCacheThread-1] INFO o.o.s.b.m.c.ApplicationDataTypeCache RequestID=null ErrorCategory=ERROR ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=200 TargetEntity=ApplicationDataTypesCache Detected changes in the data types, updating the data type cache. 2021-07-13T00:15:20.332Z [cluster1-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-07-13T00:15:20.333Z [cluster1-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.199.132:9042, scheduling retry in 4000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:15:21.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6df196b-8339-45ae-8d57-086fd94579c6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:15:21.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6df196b-8339-45ae-8d57-086fd94579c6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7689710f 2021-07-13T00:15:21.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6df196b-8339-45ae-8d57-086fd94579c6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:15:21.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6df196b-8339-45ae-8d57-086fd94579c6 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:15:21.659Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:15:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:15:22.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:15:23.071Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a593755-82c3-426e-be40-eeeac8291776 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:15:23.071Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a593755-82c3-426e-be40-eeeac8291776 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6717139b 2021-07-13T00:15:23.071Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a593755-82c3-426e-be40-eeeac8291776 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:15:23.071Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a593755-82c3-426e-be40-eeeac8291776 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:15:23.175Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6a593755-82c3-426e-be40-eeeac8291776 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:15:23.176Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6a593755-82c3-426e-be40-eeeac8291776 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:15:23.258Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.nfv.AffinityRule to dao for create 2021-07-13T00:15:24.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79279ccf-cdf5-474d-bb24-b9cdef892666 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:15:24.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79279ccf-cdf5-474d-bb24-b9cdef892666 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@79f74831 2021-07-13T00:15:24.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79279ccf-cdf5-474d-bb24-b9cdef892666 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:15:24.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79279ccf-cdf5-474d-bb24-b9cdef892666 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:15:25.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:15:25.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:15:26.185Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:15:27.072Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=79279ccf-cdf5-474d-bb24-b9cdef892666 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:15:27.073Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=79279ccf-cdf5-474d-bb24-b9cdef892666 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:15:27.228Z [cluster1-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-07-13T00:15:27.230Z [cluster1-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.199.132:9042, scheduling retry in 8000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:15:27.563Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:15:28.296Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node019afmmni64smi1tootwcixon4c214 2021-07-13T00:15:28.297Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01mqp9qhk36nzg1ahughzhifj2g217 2021-07-13T00:15:28.297Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0z7sqhlj1jih91onym71d4fxki215 2021-07-13T00:15:28.297Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0jyecgij8146nk61uw4l410ul216 2021-07-13T00:15:28.297Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01k7dksx1ubx6k1h26vzmizlbex219 2021-07-13T00:15:28.297Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0ltfsbav49ft919jhr3qd36ui0210 2021-07-13T00:15:28.297Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node05xh1r6szeef2f595690woplb231 2021-07-13T00:15:28.297Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node016po6fs5y91q8p1cxyzsahl0k194 2021-07-13T00:15:28.297Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01f5j6madmmacd65rwo8c6ninb207 2021-07-13T00:15:28.297Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01kk8oxmy8xgw8u43ylptgxrp0225 2021-07-13T00:15:28.297Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node011lpvewvl5sgwu7d1c3924m90204 2021-07-13T00:15:28.298Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node073thsoci7v1n8f821hkcr61i193 2021-07-13T00:15:28.298Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01gnz0blh8qkd814cn2xar10n2r197 2021-07-13T00:15:28.298Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0azc02q4ls8nmwsdsgf9fispk195 2021-07-13T00:15:28.298Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01n0py1618hhgl1x7ufhuuaepth202 2021-07-13T00:15:28.298Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node011bcuk09iuea21q0jpj0is2p4q196 2021-07-13T00:15:28.298Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0ssudaowfcr4m7gbvnv2qoleg192 2021-07-13T00:15:28.298Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01o82z1f5vg4x01h19c6peo19kp226 2021-07-13T00:15:28.298Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01i3f2h36aqzih1hemxyakhhdxy199 2021-07-13T00:15:28.298Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01msaf1afrmuf31oezv46nxsbbp211 2021-07-13T00:15:28.298Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0fbpcij3it50g6fw0zugecerh201 2021-07-13T00:15:28.298Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01boshprzc9mdd6mj6ut5wi2hm198 2021-07-13T00:15:28.298Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01s6a7evog7n6yrenoil1aov16218 2021-07-13T00:15:28.298Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0fob7ur5k4o19jsapt1favlnu224 2021-07-13T00:15:28.298Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01byuos2a1jkc93bx6pziw3y5v200 2021-07-13T00:15:28.299Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01t6qc58qofxamnn2lh18lljew191 2021-07-13T00:15:28.299Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0qpkzdbxl0noqpbjk8aje2tsm203 2021-07-13T00:15:28.299Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node03ja2gj379iks4pxc2vyohxys212 2021-07-13T00:15:28.299Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01gsw1z42e52ee3gp5ikhyz85a223 2021-07-13T00:15:28.299Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01n16ljmmwidan2s8ybu65npaf235 2021-07-13T00:15:28.299Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0z8he85i0qt31uoyosduu69bt209 2021-07-13T00:15:28.299Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01bi3tmcls8u7d1ua3mqfmivdr6233 2021-07-13T00:15:28.299Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01fr32ox6ylpeoylos0qoamus6221 2021-07-13T00:15:28.299Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0248azb5y53k21n3f9333cielz213 2021-07-13T00:15:28.299Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node016sf5f22kunj91bxjsah88txqo229 2021-07-13T00:15:28.299Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01hy5ug7fi6vqc1xbtz4g0smq7i206 2021-07-13T00:15:28.299Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0yd23f0qt45mm1c140kwxvwkxs230 2021-07-13T00:15:28.300Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01fjqws2mx5hsgunb08dko5mup232 2021-07-13T00:15:28.300Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0vi7lhvdlim7n1nk1qo3hr9qnq220 2021-07-13T00:15:28.300Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01fn2cpp0cax6bbm46smvbkayj222 2021-07-13T00:15:28.300Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node016xo0bzq2becn1chituy0yrkgd227 2021-07-13T00:15:28.300Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0hhq4sflhthix1fkhwnbh8evwx205 2021-07-13T00:15:28.300Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01idfupwgf4ezmw30gzgr4as9i208 2021-07-13T00:15:28.300Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node018wenxs7ru0i4v4kqlgyc26h7228 2021-07-13T00:15:28.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc4c96bc-5a1f-497b-b20b-7aedb1249366 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:15:28.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc4c96bc-5a1f-497b-b20b-7aedb1249366 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78780633 2021-07-13T00:15:28.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc4c96bc-5a1f-497b-b20b-7aedb1249366 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:15:28.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc4c96bc-5a1f-497b-b20b-7aedb1249366 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:15:28.666Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=fc4c96bc-5a1f-497b-b20b-7aedb1249366 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:15:28.666Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=fc4c96bc-5a1f-497b-b20b-7aedb1249366 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:15:29.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=133b46d2-27f5-4ff7-b04f-8144d33e3702 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:15:29.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=133b46d2-27f5-4ff7-b04f-8144d33e3702 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1a1049c2 2021-07-13T00:15:29.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=133b46d2-27f5-4ff7-b04f-8144d33e3702 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:15:29.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=133b46d2-27f5-4ff7-b04f-8144d33e3702 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:15:31.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:15:31.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:15:32.067Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=133b46d2-27f5-4ff7-b04f-8144d33e3702 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:15:32.068Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=133b46d2-27f5-4ff7-b04f-8144d33e3702 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:15:33.530Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0852080b-a2ff-49d5-9866-c86fee268671 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:15:33.530Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0852080b-a2ff-49d5-9866-c86fee268671 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@59e52c71 2021-07-13T00:15:33.530Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0852080b-a2ff-49d5-9866-c86fee268671 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:15:33.530Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0852080b-a2ff-49d5-9866-c86fee268671 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:15:33.659Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:15:34.124Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=0852080b-a2ff-49d5-9866-c86fee268671 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:15:34.124Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=0852080b-a2ff-49d5-9866-c86fee268671 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:15:35.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64fea5fb-7917-4ca8-ba8a-7f93f9cb6a33 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:15:35.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64fea5fb-7917-4ca8-ba8a-7f93f9cb6a33 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6114e0e0 2021-07-13T00:15:35.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64fea5fb-7917-4ca8-ba8a-7f93f9cb6a33 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:15:35.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64fea5fb-7917-4ca8-ba8a-7f93f9cb6a33 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:15:35.576Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=64fea5fb-7917-4ca8-ba8a-7f93f9cb6a33 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:15:35.576Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=64fea5fb-7917-4ca8-ba8a-7f93f9cb6a33 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:15:36.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46f07917-7c25-46e3-bb51-b385ce82eb35 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:15:36.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46f07917-7c25-46e3-bb51-b385ce82eb35 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@43c7029 2021-07-13T00:15:36.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46f07917-7c25-46e3-bb51-b385ce82eb35 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:15:36.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46f07917-7c25-46e3-bb51-b385ce82eb35 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:15:37.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:15:37.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:15:38.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0ca4a77-6809-4942-a73a-cdd7cc00b669 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:15:38.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0ca4a77-6809-4942-a73a-cdd7cc00b669 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c7a7735 2021-07-13T00:15:38.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0ca4a77-6809-4942-a73a-cdd7cc00b669 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:15:38.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0ca4a77-6809-4942-a73a-cdd7cc00b669 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:15:39.528Z [qtp817348612-16] WARN o.j.d.l.c.ConsistentKeyLocker RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Temporary storage failure while checking locks org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.getSliceWithRetries(ConsistentKeyLocker.java:464) at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSingleLock(ConsistentKeyLocker.java:426) at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSingleLock(ConsistentKeyLocker.java:118) at org.janusgraph.diskstorage.locking.AbstractLocker.checkLocks(AbstractLocker.java:351) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.checkAllLocks(ExpectedValueCheckingTransaction.java:175) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.prepareForMutations(ExpectedValueCheckingTransaction.java:154) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStoreManager.mutateMany(ExpectedValueCheckingStoreManager.java:72) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:94) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:91) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:91) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:139) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:196) at org.janusgraph.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:134) at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:733) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1379) at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.doCommit(JanusGraphBlueprintsGraph.java:298) at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:104) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.commit(JanusGraphClient.java:250) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.commit(JanusGraphGenericDao.java:74) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.handleTransactionCommitRollback(JanusGraphGenericDao.java:87) at org.openecomp.sdc.be.model.operations.impl.GroupTypeOperation.getGroupTypeByCriteria(GroupTypeOperation.java:346) at org.openecomp.sdc.be.model.operations.impl.GroupTypeOperation.getLatestGroupTypeByType(GroupTypeOperation.java:322) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.validatePolicyType(PolicyTypeImportManager.java:103) at org.openecomp.sdc.be.components.impl.CommonImportManager.handleType(CommonImportManager.java:305) at org.openecomp.sdc.be.components.impl.CommonImportManager.createElementTypesByDao(CommonImportManager.java:268) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.upsertPolicyTypesByDao(PolicyTypeImportManager.java:74) at fj.data.Either$LeftProjection.bind(Either.java:287) at org.openecomp.sdc.be.components.impl.CommonImportManager.createElementTypes(CommonImportManager.java:392) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.createPolicyTypes(PolicyTypeImportManager.java:65) at org.openecomp.sdc.be.servlets.TypesUploadServlet.lambda$createPolicyTypes$8(TypesUploadServlet.java:334) at org.openecomp.sdc.be.servlets.TypesUploadServlet.buildStatusForElementTypeCreate(TypesUploadServlet.java:343) at org.openecomp.sdc.be.servlets.TypesUploadServlet.createPolicyTypes(TypesUploadServlet.java:335) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadTypesWithMetaData(TypesUploadServlet.java:287) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadPolicyTypes_aroundBody12(TypesUploadServlet.java:228) at org.openecomp.sdc.be.servlets.TypesUploadServlet$AjcClosure13.run(TypesUploadServlet.java:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:212) at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) at com.jcabi.aspects.aj.MethodLogger.wrapClass(MethodLogger.java:136) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadPolicyTypes(TypesUploadServlet.java:227) at org.openecomp.sdc.be.servlets.TypesUploadServlet$$FastClassBySpringCGLIB$$893260aa.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at org.openecomp.sdc.be.servlets.TypesUploadServlet$$EnhancerBySpringCGLIB$$8228831e.uploadPolicyTypes() at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1411) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:226) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.togglz.servlet.TogglzFilter.doFilter(TogglzFilter.java:100) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.ReqValidationFilter.doFilter(ReqValidationFilter.java:72) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.GatewayFilter.doFilter(GatewayFilter.java:71) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1630) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:501) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ALL (3 responses were required but only 2 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ALL (3 responses were required but only 2 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ALL (3 responses were required but only 2 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:15:43.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:15:43.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:15:46.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:15:49.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:15:50.758Z [qtp817348612-16] INFO o.j.d.util.BackendOperation RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Temporary exception during backend operation [CacheMutation]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateManyUnlogged(CQLStoreManager.java:504) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateMany(CQLStoreManager.java:435) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStoreManager.mutateMany(ExpectedValueCheckingStoreManager.java:76) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:94) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:91) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:91) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:139) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:196) at org.janusgraph.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:134) at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:733) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1379) at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.doCommit(JanusGraphBlueprintsGraph.java:298) at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:104) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.commit(JanusGraphClient.java:250) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.commit(JanusGraphGenericDao.java:74) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.handleTransactionCommitRollback(JanusGraphGenericDao.java:87) at org.openecomp.sdc.be.model.operations.impl.GroupTypeOperation.getGroupTypeByCriteria(GroupTypeOperation.java:346) at org.openecomp.sdc.be.model.operations.impl.GroupTypeOperation.getLatestGroupTypeByType(GroupTypeOperation.java:322) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.validatePolicyType(PolicyTypeImportManager.java:103) at org.openecomp.sdc.be.components.impl.CommonImportManager.handleType(CommonImportManager.java:305) at org.openecomp.sdc.be.components.impl.CommonImportManager.createElementTypesByDao(CommonImportManager.java:268) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.upsertPolicyTypesByDao(PolicyTypeImportManager.java:74) at fj.data.Either$LeftProjection.bind(Either.java:287) at org.openecomp.sdc.be.components.impl.CommonImportManager.createElementTypes(CommonImportManager.java:392) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.createPolicyTypes(PolicyTypeImportManager.java:65) at org.openecomp.sdc.be.servlets.TypesUploadServlet.lambda$createPolicyTypes$8(TypesUploadServlet.java:334) at org.openecomp.sdc.be.servlets.TypesUploadServlet.buildStatusForElementTypeCreate(TypesUploadServlet.java:343) at org.openecomp.sdc.be.servlets.TypesUploadServlet.createPolicyTypes(TypesUploadServlet.java:335) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadTypesWithMetaData(TypesUploadServlet.java:287) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadPolicyTypes_aroundBody12(TypesUploadServlet.java:228) at org.openecomp.sdc.be.servlets.TypesUploadServlet$AjcClosure13.run(TypesUploadServlet.java:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:212) at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) at com.jcabi.aspects.aj.MethodLogger.wrapClass(MethodLogger.java:136) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadPolicyTypes(TypesUploadServlet.java:227) at org.openecomp.sdc.be.servlets.TypesUploadServlet$$FastClassBySpringCGLIB$$893260aa.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at org.openecomp.sdc.be.servlets.TypesUploadServlet$$EnhancerBySpringCGLIB$$8228831e.uploadPolicyTypes() at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1411) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:226) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.togglz.servlet.TogglzFilter.doFilter(TogglzFilter.java:100) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.ReqValidationFilter.doFilter(ReqValidationFilter.java:72) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.GatewayFilter.doFilter(GatewayFilter.java:71) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1630) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:501) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:113) at com.datastax.driver.core.Responses$Error.asException(Responses.java:167) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:87) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:15:51.843Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:15:52.146Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc04280b-9bcc-43ac-b903-70fea19ed0d8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:15:52.146Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc04280b-9bcc-43ac-b903-70fea19ed0d8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6c0c566f 2021-07-13T00:15:52.146Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc04280b-9bcc-43ac-b903-70fea19ed0d8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:15:52.146Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc04280b-9bcc-43ac-b903-70fea19ed0d8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:15:52.918Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:15:52.929Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:15:55.172Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=bc04280b-9bcc-43ac-b903-70fea19ed0d8 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out 2021-07-13T00:15:55.172Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=bc04280b-9bcc-43ac-b903-70fea19ed0d8 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:15:55.173Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=bc04280b-9bcc-43ac-b903-70fea19ed0d8 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:15:55.173Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=bc04280b-9bcc-43ac-b903-70fea19ed0d8 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:15:55.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:15:55.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:15:58.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:16:01.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:16:04.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:16:07.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:16:07.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:16:07.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:16:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:16:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:16:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:16:10.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:16:13.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:16:14.965Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.nfv.AntiAffinityRule to dao for create 2021-07-13T00:16:15.266Z [cluster1-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.199.132:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.199.132:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:16:15.267Z [cluster1-timeouter-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.199.132:9042 com.datastax.driver.core.exceptions.ConnectionException: [cassandra.onap/10.242.199.132:9042] Pool was closed during initialization at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:173) at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:158) at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1080) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.google.common.util.concurrent.CollectionFuture.handleAllCompleted(CollectionFuture.java:69) at com.google.common.util.concurrent.AggregateFuture.processCompleted(AggregateFuture.java:282) at com.google.common.util.concurrent.AggregateFuture.decrementCountAndMaybeComplete(AggregateFuture.java:264) at com.google.common.util.concurrent.AggregateFuture.access$200(AggregateFuture.java:42) at com.google.common.util.concurrent.AggregateFuture$1.run(AggregateFuture.java:147) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:100) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:16:15.267Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95579a6f-f5c4-4f7d-8f8f-2ee1e853ee83 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:16:15.267Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95579a6f-f5c4-4f7d-8f8f-2ee1e853ee83 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4d8338a4 2021-07-13T00:16:15.267Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95579a6f-f5c4-4f7d-8f8f-2ee1e853ee83 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:16:15.267Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95579a6f-f5c4-4f7d-8f8f-2ee1e853ee83 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:16:16.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:16:18.281Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=95579a6f-f5c4-4f7d-8f8f-2ee1e853ee83 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out 2021-07-13T00:16:18.281Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=95579a6f-f5c4-4f7d-8f8f-2ee1e853ee83 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:16:19.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=30257062-0235-41ff-966b-749fc8469ae7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:16:19.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=30257062-0235-41ff-966b-749fc8469ae7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@cbeacee 2021-07-13T00:16:19.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=30257062-0235-41ff-966b-749fc8469ae7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:16:19.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=30257062-0235-41ff-966b-749fc8469ae7 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:16:19.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:16:22.387Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=30257062-0235-41ff-966b-749fc8469ae7 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out 2021-07-13T00:16:22.388Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=30257062-0235-41ff-966b-749fc8469ae7 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:16:22.388Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=30257062-0235-41ff-966b-749fc8469ae7 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:16:22.388Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=30257062-0235-41ff-966b-749fc8469ae7 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:16:22.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:16:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:16:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:16:23.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2319bbf2-76e6-4190-884e-eb124d93c4c2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:16:23.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2319bbf2-76e6-4190-884e-eb124d93c4c2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3fd5388b 2021-07-13T00:16:23.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2319bbf2-76e6-4190-884e-eb124d93c4c2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:16:23.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2319bbf2-76e6-4190-884e-eb124d93c4c2 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:16:24.986Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:16:25.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:16:26.295Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=2319bbf2-76e6-4190-884e-eb124d93c4c2 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:16:26.295Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=2319bbf2-76e6-4190-884e-eb124d93c4c2 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:16:27.169Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8315bc88-af8e-41a8-b92a-ebd6351e87c2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:16:27.170Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8315bc88-af8e-41a8-b92a-ebd6351e87c2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3d6e9fe9 2021-07-13T00:16:27.170Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8315bc88-af8e-41a8-b92a-ebd6351e87c2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:16:27.170Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8315bc88-af8e-41a8-b92a-ebd6351e87c2 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:16:28.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:16:30.233Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=8315bc88-af8e-41a8-b92a-ebd6351e87c2 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:16:30.233Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=8315bc88-af8e-41a8-b92a-ebd6351e87c2 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:16:31.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5b68023-90d5-4694-a1a1-256e1597d0d2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:16:31.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5b68023-90d5-4694-a1a1-256e1597d0d2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7b246ac7 2021-07-13T00:16:31.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5b68023-90d5-4694-a1a1-256e1597d0d2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:16:31.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5b68023-90d5-4694-a1a1-256e1597d0d2 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:16:31.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:16:33.760Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:16:34.212Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=e5b68023-90d5-4694-a1a1-256e1597d0d2 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:16:34.213Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=e5b68023-90d5-4694-a1a1-256e1597d0d2 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:16:35.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=946d9019-bffc-41b1-bccc-c16ca157b986 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:16:35.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=946d9019-bffc-41b1-bccc-c16ca157b986 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2107424 2021-07-13T00:16:35.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=946d9019-bffc-41b1-bccc-c16ca157b986 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:16:35.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=946d9019-bffc-41b1-bccc-c16ca157b986 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:16:36.378Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=946d9019-bffc-41b1-bccc-c16ca157b986 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:16:36.378Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=946d9019-bffc-41b1-bccc-c16ca157b986 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:16:37.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c600a29-51e7-439a-bcf1-3d5df5b96b54 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:16:37.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c600a29-51e7-439a-bcf1-3d5df5b96b54 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2f581bc3 2021-07-13T00:16:37.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c600a29-51e7-439a-bcf1-3d5df5b96b54 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:16:37.259Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c600a29-51e7-439a-bcf1-3d5df5b96b54 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:16:37.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:16:37.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:16:38.241Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.199.132:9042: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-07-13T00:16:38.242Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.199.132:9042, scheduling retry in 2000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.199.132:9042: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:16:38.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b6dcdad-6319-47d3-971e-f10c261640bc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:16:38.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b6dcdad-6319-47d3-971e-f10c261640bc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@25c13ef8 2021-07-13T00:16:38.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b6dcdad-6319-47d3-971e-f10c261640bc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:16:38.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b6dcdad-6319-47d3-971e-f10c261640bc PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:16:39.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5452108a-8dd0-40c3-b14f-18c05bc233e9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:16:39.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5452108a-8dd0-40c3-b14f-18c05bc233e9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2471cf30 2021-07-13T00:16:39.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5452108a-8dd0-40c3-b14f-18c05bc233e9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:16:39.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5452108a-8dd0-40c3-b14f-18c05bc233e9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:16:42.561Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=5452108a-8dd0-40c3-b14f-18c05bc233e9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:16:42.562Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=5452108a-8dd0-40c3-b14f-18c05bc233e9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:16:43.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:16:43.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:16:43.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ae7bcea-f628-4351-b230-8488df2f6f0c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:16:43.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ae7bcea-f628-4351-b230-8488df2f6f0c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@42c8f163 2021-07-13T00:16:43.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ae7bcea-f628-4351-b230-8488df2f6f0c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:16:43.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ae7bcea-f628-4351-b230-8488df2f6f0c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:16:45.422Z [ApplicationDataTypeCacheThread-1] INFO o.j.d.util.BackendOperation RequestID=null ErrorCategory=ERROR ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=200 TargetEntity=ApplicationDataTypesCache Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Element.keys(Element.java:69) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getProperties(JanusGraphGenericDao.java:1491) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getChildrenNodes(JanusGraphGenericDao.java:1198) at org.openecomp.sdc.be.dao.janusgraph.HealingJanusGraphGenericDao.getChildrenNodes(HealingJanusGraphGenericDao.java:62) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getChildrenNodes(JanusGraphGenericDao.java:1243) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.findPropertiesOfNode(PropertyOperation.java:344) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.fillProperties(PropertyOperation.java:1272) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAndAddDataTypeByUid(PropertyOperation.java:1519) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAllDataTypes(PropertyOperation.java:1474) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.findAllDataTypesEager(ApplicationDataTypeCache.java:294) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.refreshDataTypesCache(ApplicationDataTypeCache.java:276) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.run(ApplicationDataTypeCache.java:202) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:16:45.424Z [qtp817348612-16] INFO o.j.d.util.BackendOperation RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Temporary exception during backend operation [CacheMutation]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateManyUnlogged(CQLStoreManager.java:504) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateMany(CQLStoreManager.java:435) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStoreManager.mutateMany(ExpectedValueCheckingStoreManager.java:76) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:94) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:91) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:91) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:139) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:196) at org.janusgraph.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:134) at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:733) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1379) at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.doCommit(JanusGraphBlueprintsGraph.java:298) at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:104) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.commit(JanusGraphClient.java:250) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.commit(JanusGraphGenericDao.java:74) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.handleTransactionCommitRollback(JanusGraphGenericDao.java:87) at org.openecomp.sdc.be.model.operations.impl.GroupTypeOperation.getGroupTypeByCriteria(GroupTypeOperation.java:346) at org.openecomp.sdc.be.model.operations.impl.GroupTypeOperation.getLatestGroupTypeByType(GroupTypeOperation.java:322) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.validatePolicyType(PolicyTypeImportManager.java:103) at org.openecomp.sdc.be.components.impl.CommonImportManager.handleType(CommonImportManager.java:305) at org.openecomp.sdc.be.components.impl.CommonImportManager.createElementTypesByDao(CommonImportManager.java:268) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.upsertPolicyTypesByDao(PolicyTypeImportManager.java:74) at fj.data.Either$LeftProjection.bind(Either.java:287) at org.openecomp.sdc.be.components.impl.CommonImportManager.createElementTypes(CommonImportManager.java:392) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.createPolicyTypes(PolicyTypeImportManager.java:65) at org.openecomp.sdc.be.servlets.TypesUploadServlet.lambda$createPolicyTypes$8(TypesUploadServlet.java:334) at org.openecomp.sdc.be.servlets.TypesUploadServlet.buildStatusForElementTypeCreate(TypesUploadServlet.java:343) at org.openecomp.sdc.be.servlets.TypesUploadServlet.createPolicyTypes(TypesUploadServlet.java:335) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadTypesWithMetaData(TypesUploadServlet.java:287) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadPolicyTypes_aroundBody12(TypesUploadServlet.java:228) at org.openecomp.sdc.be.servlets.TypesUploadServlet$AjcClosure13.run(TypesUploadServlet.java:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:212) at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) at com.jcabi.aspects.aj.MethodLogger.wrapClass(MethodLogger.java:136) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadPolicyTypes(TypesUploadServlet.java:227) at org.openecomp.sdc.be.servlets.TypesUploadServlet$$FastClassBySpringCGLIB$$893260aa.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at org.openecomp.sdc.be.servlets.TypesUploadServlet$$EnhancerBySpringCGLIB$$8228831e.uploadPolicyTypes() at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1411) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:226) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.togglz.servlet.TogglzFilter.doFilter(TogglzFilter.java:100) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.ReqValidationFilter.doFilter(ReqValidationFilter.java:72) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.GatewayFilter.doFilter(GatewayFilter.java:71) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1630) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:501) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:113) at com.datastax.driver.core.Responses$Error.asException(Responses.java:167) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:308) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:422) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:87) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 26 common frames omitted 2021-07-13T00:16:46.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:16:46.516Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=4ae7bcea-f628-4351-b230-8488df2f6f0c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:16:46.516Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=4ae7bcea-f628-4351-b230-8488df2f6f0c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:16:47.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c41f53a4-392d-4e5b-8418-0aeecaba7efe PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:16:47.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c41f53a4-392d-4e5b-8418-0aeecaba7efe PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1bbb158 2021-07-13T00:16:47.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c41f53a4-392d-4e5b-8418-0aeecaba7efe PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:16:47.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c41f53a4-392d-4e5b-8418-0aeecaba7efe PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:16:48.374Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c41f53a4-392d-4e5b-8418-0aeecaba7efe ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:16:48.375Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=c41f53a4-392d-4e5b-8418-0aeecaba7efe ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:16:49.370Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44fd55cc-d442-47c3-91e7-ca381e974c4d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:16:49.370Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44fd55cc-d442-47c3-91e7-ca381e974c4d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2edf59e9 2021-07-13T00:16:49.370Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44fd55cc-d442-47c3-91e7-ca381e974c4d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:16:49.370Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44fd55cc-d442-47c3-91e7-ca381e974c4d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:16:49.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:16:50.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f49abe7e-0697-4c7d-b138-60cbeb068cdf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:16:50.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f49abe7e-0697-4c7d-b138-60cbeb068cdf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@83f2664 2021-07-13T00:16:50.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f49abe7e-0697-4c7d-b138-60cbeb068cdf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:16:50.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f49abe7e-0697-4c7d-b138-60cbeb068cdf PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:16:51.226Z [pool-8-thread-1] WARN o.j.diskstorage.log.kcvs.KCVSLog Could not read messages for timestamp [2021-07-13T00:16:35.760668Z] (this read will be retried) org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:56) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:726) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:100) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) ... 8 common frames omitted Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:792) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:789) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) ... 9 common frames omitted Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:308) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:422) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 26 common frames omitted 2021-07-13T00:16:51.226Z [JanusGraph-Health-Check-Thread] INFO o.j.d.util.BackendOperation Temporary exception during backend operation [VertexIndexQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:399) at org.janusgraph.diskstorage.BackendTransaction$5.call(BackendTransaction.java:396) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.indexQuery(BackendTransaction.java:396) at org.janusgraph.graphdb.query.graph.MultiKeySliceQuery.execute(MultiKeySliceQuery.java:51) at org.janusgraph.graphdb.database.IndexSerializer.query(IndexSerializer.java:512) at org.janusgraph.graphdb.util.SubqueryIterator.(SubqueryIterator.java:66) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1295) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute(StandardJanusGraphTx.java:1190) at org.janusgraph.graphdb.query.QueryProcessor$LimitAdjustingIterator.getNewIterator(QueryProcessor.java:194) at org.janusgraph.graphdb.query.LimitAdjustingIterator.hasNext(LimitAdjustingIterator.java:68) at org.janusgraph.graphdb.query.ResultSetIterator.nextInternal(ResultSetIterator.java:54) at org.janusgraph.graphdb.query.ResultSetIterator.(ResultSetIterator.java:44) at org.janusgraph.graphdb.query.QueryProcessor.iterator(QueryProcessor.java:66) at com.google.common.collect.Iterables$4.iterator(Iterables.java:559) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:340) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient$HealthCheckTask.call(JanusGraphClient.java:336) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ... 4 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:308) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:422) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 26 common frames omitted 2021-07-13T00:16:51.227Z [ApplicationDataTypeCacheThread-1] ERROR o.o.s.b.m.c.ApplicationDataTypeCache RequestID=null ErrorCategory=ERROR ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=900 TargetEntity=null Failed to run refresh data types cache job org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:56) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Element.keys(Element.java:69) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getProperties(JanusGraphGenericDao.java:1491) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getChildrenNodes(JanusGraphGenericDao.java:1198) at org.openecomp.sdc.be.dao.janusgraph.HealingJanusGraphGenericDao.getChildrenNodes(HealingJanusGraphGenericDao.java:62) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getChildrenNodes(JanusGraphGenericDao.java:1243) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.findPropertiesOfNode(PropertyOperation.java:344) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.fillProperties(PropertyOperation.java:1272) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAndAddDataTypeByUid(PropertyOperation.java:1519) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAllDataTypes(PropertyOperation.java:1474) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.findAllDataTypesEager(ApplicationDataTypeCache.java:294) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.refreshDataTypesCache(ApplicationDataTypeCache.java:276) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.run(ApplicationDataTypeCache.java:202) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT10S at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:100) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) ... 32 common frames omitted Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) ... 33 common frames omitted Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:16:51.227Z [ApplicationDataTypeCacheThread-1] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=900 TargetEntity=ApplicationDataTypesCache Unexpected error. Description: Failed to run refresh data types cache job 2021-07-13T00:16:52.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:16:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:16:52.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:16:53.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a2a3914-0067-4d4b-9454-2f93c9fe3248 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:16:53.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a2a3914-0067-4d4b-9454-2f93c9fe3248 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5452774a 2021-07-13T00:16:53.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a2a3914-0067-4d4b-9454-2f93c9fe3248 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:16:53.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a2a3914-0067-4d4b-9454-2f93c9fe3248 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:16:54.015Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-07-13T00:16:54.016Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.199.132:9042, scheduling retry in 4000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:16:54.017Z [cluster2-reconnection-0] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-07-13T00:16:54.019Z [cluster2-reconnection-0] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.199.132:9042, scheduling retry in 30000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:16:54.020Z [qtp817348612-16] INFO o.j.d.util.BackendOperation RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Temporary exception during backend operation [CacheMutation]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateManyUnlogged(CQLStoreManager.java:504) at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateMany(CQLStoreManager.java:435) at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStoreManager.mutateMany(ExpectedValueCheckingStoreManager.java:76) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:94) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:91) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:91) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:139) at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:196) at org.janusgraph.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:134) at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:733) at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1379) at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.doCommit(JanusGraphBlueprintsGraph.java:298) at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:104) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphClient.commit(JanusGraphClient.java:250) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.commit(JanusGraphGenericDao.java:74) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.handleTransactionCommitRollback(JanusGraphGenericDao.java:87) at org.openecomp.sdc.be.model.operations.impl.GroupTypeOperation.getGroupTypeByCriteria(GroupTypeOperation.java:346) at org.openecomp.sdc.be.model.operations.impl.GroupTypeOperation.getLatestGroupTypeByType(GroupTypeOperation.java:322) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.validatePolicyType(PolicyTypeImportManager.java:103) at org.openecomp.sdc.be.components.impl.CommonImportManager.handleType(CommonImportManager.java:305) at org.openecomp.sdc.be.components.impl.CommonImportManager.createElementTypesByDao(CommonImportManager.java:268) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.upsertPolicyTypesByDao(PolicyTypeImportManager.java:74) at fj.data.Either$LeftProjection.bind(Either.java:287) at org.openecomp.sdc.be.components.impl.CommonImportManager.createElementTypes(CommonImportManager.java:392) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.createPolicyTypes(PolicyTypeImportManager.java:65) at org.openecomp.sdc.be.servlets.TypesUploadServlet.lambda$createPolicyTypes$8(TypesUploadServlet.java:334) at org.openecomp.sdc.be.servlets.TypesUploadServlet.buildStatusForElementTypeCreate(TypesUploadServlet.java:343) at org.openecomp.sdc.be.servlets.TypesUploadServlet.createPolicyTypes(TypesUploadServlet.java:335) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadTypesWithMetaData(TypesUploadServlet.java:287) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadPolicyTypes_aroundBody12(TypesUploadServlet.java:228) at org.openecomp.sdc.be.servlets.TypesUploadServlet$AjcClosure13.run(TypesUploadServlet.java:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:212) at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) at com.jcabi.aspects.aj.MethodLogger.wrapClass(MethodLogger.java:136) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadPolicyTypes(TypesUploadServlet.java:227) at org.openecomp.sdc.be.servlets.TypesUploadServlet$$FastClassBySpringCGLIB$$893260aa.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at org.openecomp.sdc.be.servlets.TypesUploadServlet$$EnhancerBySpringCGLIB$$8228831e.uploadPolicyTypes() at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1411) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:226) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.togglz.servlet.TogglzFilter.doFilter(TogglzFilter.java:100) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.ReqValidationFilter.doFilter(ReqValidationFilter.java:72) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.GatewayFilter.doFilter(GatewayFilter.java:71) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1630) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:501) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:113) at com.datastax.driver.core.Responses$Error.asException(Responses.java:167) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during SIMPLE write query at consistency QUORUM (2 replica were required but only 1 acknowledged the write) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:87) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:16:54.023Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.199.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:16:55.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:16:55.468Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ff0f545-ab21-49af-80d7-6b1b10f12b68 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:16:55.468Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ff0f545-ab21-49af-80d7-6b1b10f12b68 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@309fa2b6 2021-07-13T00:16:55.468Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ff0f545-ab21-49af-80d7-6b1b10f12b68 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:16:55.468Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ff0f545-ab21-49af-80d7-6b1b10f12b68 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:16:58.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:16:58.474Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=8ff0f545-ab21-49af-80d7-6b1b10f12b68 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:16:58.474Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=8ff0f545-ab21-49af-80d7-6b1b10f12b68 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:16:58.474Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=8ff0f545-ab21-49af-80d7-6b1b10f12b68 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:16:58.474Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=8ff0f545-ab21-49af-80d7-6b1b10f12b68 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:16:59.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3549362-eeb2-48d2-873d-38fee9cf7531 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:16:59.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3549362-eeb2-48d2-873d-38fee9cf7531 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64dbb55d 2021-07-13T00:16:59.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3549362-eeb2-48d2-873d-38fee9cf7531 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:16:59.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3549362-eeb2-48d2-873d-38fee9cf7531 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:17:01.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:17:02.627Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=d3549362-eeb2-48d2-873d-38fee9cf7531 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:17:02.628Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=d3549362-eeb2-48d2-873d-38fee9cf7531 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:17:03.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b720d73-3814-48cd-a800-4cee3409c77a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:17:03.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b720d73-3814-48cd-a800-4cee3409c77a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@52097e7d 2021-07-13T00:17:03.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b720d73-3814-48cd-a800-4cee3409c77a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:17:03.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b720d73-3814-48cd-a800-4cee3409c77a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:17:04.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:17:06.172Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.nfv.NsAffinityRule to dao for create 2021-07-13T00:17:06.609Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=5b720d73-3814-48cd-a800-4cee3409c77a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:17:06.610Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=5b720d73-3814-48cd-a800-4cee3409c77a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:17:07.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4fc7cc50-4c10-4da3-a637-ae3eabadc57e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:17:07.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4fc7cc50-4c10-4da3-a637-ae3eabadc57e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2686c361 2021-07-13T00:17:07.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4fc7cc50-4c10-4da3-a637-ae3eabadc57e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:17:07.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4fc7cc50-4c10-4da3-a637-ae3eabadc57e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:17:07.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:17:07.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:17:07.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:17:08.474Z [cluster2-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.199.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:17:08.475Z [cluster2-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.199.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:17:08.476Z [cluster2-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.199.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:17:08.476Z [cluster2-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.199.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:17:08.516Z [cluster2-nio-worker-1] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.199.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:17:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:17:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:17:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:17:09.378Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=4fc7cc50-4c10-4da3-a637-ae3eabadc57e ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:17:09.378Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=4fc7cc50-4c10-4da3-a637-ae3eabadc57e ErrorCategory=ERROR ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:17:10.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:17:10.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ed6c106-d9be-45af-8748-b36206bd4a51 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:17:10.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ed6c106-d9be-45af-8748-b36206bd4a51 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1ffe83ae 2021-07-13T00:17:10.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ed6c106-d9be-45af-8748-b36206bd4a51 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:17:10.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8ed6c106-d9be-45af-8748-b36206bd4a51 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:17:13.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:17:13.673Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=8ed6c106-d9be-45af-8748-b36206bd4a51 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:17:13.673Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=8ed6c106-d9be-45af-8748-b36206bd4a51 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:17:13.673Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=8ed6c106-d9be-45af-8748-b36206bd4a51 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:17:13.673Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=8ed6c106-d9be-45af-8748-b36206bd4a51 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:17:14.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4dc0b933-fe9f-4ae7-8af5-c0e9f7b09fff PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:17:14.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4dc0b933-fe9f-4ae7-8af5-c0e9f7b09fff PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@773c86f5 2021-07-13T00:17:14.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4dc0b933-fe9f-4ae7-8af5-c0e9f7b09fff PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:17:14.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4dc0b933-fe9f-4ae7-8af5-c0e9f7b09fff PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:17:16.465Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=4dc0b933-fe9f-4ae7-8af5-c0e9f7b09fff ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:17:16.465Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=4dc0b933-fe9f-4ae7-8af5-c0e9f7b09fff ErrorCategory=ERROR ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:17:16.613Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:17:17.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a0d484b-066a-420b-86a9-4cd1eb22e68c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:17:17.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a0d484b-066a-420b-86a9-4cd1eb22e68c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e397c97 2021-07-13T00:17:17.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a0d484b-066a-420b-86a9-4cd1eb22e68c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:17:17.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a0d484b-066a-420b-86a9-4cd1eb22e68c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:17:22.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:17:22.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:17:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:17:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:17:24.659Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:17:25.805Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:17:28.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:17:28.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:17:31.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:17:34.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:17:37.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:17:37.917Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:17:37.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:17:37.934Z [cluster1-nio-worker-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.199.132:9042 com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:215) at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:242) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:17:37.935Z [cluster1-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.199.132:9042 com.datastax.driver.core.exceptions.ConnectionException: [cassandra.onap/10.242.199.132:9042] Pool was closed during initialization at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:173) at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:158) at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1080) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.google.common.util.concurrent.CollectionFuture.handleAllCompleted(CollectionFuture.java:69) at com.google.common.util.concurrent.AggregateFuture.processCompleted(AggregateFuture.java:282) at com.google.common.util.concurrent.AggregateFuture.decrementCountAndMaybeComplete(AggregateFuture.java:264) at com.google.common.util.concurrent.AggregateFuture.access$200(AggregateFuture.java:42) at com.google.common.util.concurrent.AggregateFuture$1.run(AggregateFuture.java:147) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:100) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:215) at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:242) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:17:37.935Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f046ee2b-8a49-424d-b9c2-2ba5a478ff78 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:17:37.935Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f046ee2b-8a49-424d-b9c2-2ba5a478ff78 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@65b16d0d 2021-07-13T00:17:37.935Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f046ee2b-8a49-424d-b9c2-2ba5a478ff78 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:17:37.936Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f046ee2b-8a49-424d-b9c2-2ba5a478ff78 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:17:40.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:17:40.944Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=f046ee2b-8a49-424d-b9c2-2ba5a478ff78 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:17:40.945Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=f046ee2b-8a49-424d-b9c2-2ba5a478ff78 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:17:40.945Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f046ee2b-8a49-424d-b9c2-2ba5a478ff78 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:17:40.945Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f046ee2b-8a49-424d-b9c2-2ba5a478ff78 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:17:42.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fc3019d-23e5-4048-abfa-9f81c39e7759 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:17:42.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fc3019d-23e5-4048-abfa-9f81c39e7759 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@76294460 2021-07-13T00:17:42.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fc3019d-23e5-4048-abfa-9f81c39e7759 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:17:42.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fc3019d-23e5-4048-abfa-9f81c39e7759 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:17:43.326Z [qtp817348612-16] INFO o.j.d.util.BackendOperation RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Element.keys(Element.java:69) at org.openecomp.sdc.be.dao.jsongraph.JanusGraphDao.getVertexProperties(JanusGraphDao.java:335) at org.openecomp.sdc.be.dao.jsongraph.JanusGraphDao.parseVertexProperties(JanusGraphDao.java:267) at org.openecomp.sdc.be.dao.jsongraph.JanusGraphDao.createAndFill(JanusGraphDao.java:261) at org.openecomp.sdc.be.dao.jsongraph.JanusGraphDao.lambda$getAdjacentVertices$3(JanusGraphDao.java:673) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source) at java.base/java.util.stream.ReferencePipeline$Head.forEach(Unknown Source) at org.openecomp.sdc.be.dao.jsongraph.JanusGraphDao.getAdjacentVertices(JanusGraphDao.java:672) at org.openecomp.sdc.be.dao.jsongraph.JanusGraphDao.getChildrenVertices(JanusGraphDao.java:607) at org.openecomp.sdc.be.dao.jsongraph.HealingJanusGraphDao.getChildrenVertices(HealingJanusGraphDao.java:44) at org.openecomp.sdc.be.dao.jsongraph.JanusGraphDao.getChildVertex(JanusGraphDao.java:555) at org.openecomp.sdc.be.model.jsonjanusgraph.operations.BaseOperation.getDataVertex(BaseOperation.java:216) at org.openecomp.sdc.be.model.jsonjanusgraph.operations.BaseOperation.getDataAndVertexFromGraph(BaseOperation.java:204) at org.openecomp.sdc.be.model.jsonjanusgraph.operations.BaseOperation.getDataFromGraph(BaseOperation.java:193) at org.openecomp.sdc.be.model.jsonjanusgraph.operations.NodeTypeOperation.setResourceAttributesFromGraph(NodeTypeOperation.java:364) at org.openecomp.sdc.be.model.jsonjanusgraph.operations.NodeTypeOperation.getToscaElement(NodeTypeOperation.java:199) at org.openecomp.sdc.be.model.jsonjanusgraph.operations.ToscaOperationFacade.getToscaElementByOperation(ToscaOperationFacade.java:254) at org.openecomp.sdc.be.model.jsonjanusgraph.operations.ToscaOperationFacade.getLatestByName(ToscaOperationFacade.java:768) at org.openecomp.sdc.be.model.jsonjanusgraph.operations.ToscaOperationFacade.getLatestByName(ToscaOperationFacade.java:737) at org.openecomp.sdc.be.model.jsonjanusgraph.operations.ToscaOperationFacade.getLatestByName(ToscaOperationFacade.java:773) at org.openecomp.sdc.be.model.jsonjanusgraph.operations.ToscaOperationFacade.getLatestByToscaResourceName(ToscaOperationFacade.java:352) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.validatePolicyType(PolicyTypeImportManager.java:100) at org.openecomp.sdc.be.components.impl.CommonImportManager.handleType(CommonImportManager.java:305) at org.openecomp.sdc.be.components.impl.CommonImportManager.createElementTypesByDao(CommonImportManager.java:268) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.upsertPolicyTypesByDao(PolicyTypeImportManager.java:74) at fj.data.Either$LeftProjection.bind(Either.java:287) at org.openecomp.sdc.be.components.impl.CommonImportManager.createElementTypes(CommonImportManager.java:392) at org.openecomp.sdc.be.components.impl.PolicyTypeImportManager.createPolicyTypes(PolicyTypeImportManager.java:65) at org.openecomp.sdc.be.servlets.TypesUploadServlet.lambda$createPolicyTypes$8(TypesUploadServlet.java:334) at org.openecomp.sdc.be.servlets.TypesUploadServlet.buildStatusForElementTypeCreate(TypesUploadServlet.java:343) at org.openecomp.sdc.be.servlets.TypesUploadServlet.createPolicyTypes(TypesUploadServlet.java:335) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadTypesWithMetaData(TypesUploadServlet.java:287) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadPolicyTypes_aroundBody12(TypesUploadServlet.java:228) at org.openecomp.sdc.be.servlets.TypesUploadServlet$AjcClosure13.run(TypesUploadServlet.java:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:212) at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) at com.jcabi.aspects.aj.MethodLogger.wrapClass(MethodLogger.java:136) at org.openecomp.sdc.be.servlets.TypesUploadServlet.uploadPolicyTypes(TypesUploadServlet.java:227) at org.openecomp.sdc.be.servlets.TypesUploadServlet$$FastClassBySpringCGLIB$$893260aa.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at org.openecomp.sdc.be.servlets.TypesUploadServlet$$EnhancerBySpringCGLIB$$8228831e.uploadPolicyTypes() at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1411) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:226) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.togglz.servlet.TogglzFilter.doFilter(TogglzFilter.java:100) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.ReqValidationFilter.doFilter(ReqValidationFilter.java:72) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638) at org.openecomp.sdc.be.filters.GatewayFilter.doFilter(GatewayFilter.java:71) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1630) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:501) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 0 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 0 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 0 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:17:43.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:17:43.629Z [cluster2-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-07-13T00:17:43.630Z [cluster2-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.199.132:9042, scheduling retry in 30000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:17:45.105Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=6fc3019d-23e5-4048-abfa-9f81c39e7759 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:17:45.105Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=6fc3019d-23e5-4048-abfa-9f81c39e7759 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:17:45.106Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6fc3019d-23e5-4048-abfa-9f81c39e7759 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:17:45.106Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=6fc3019d-23e5-4048-abfa-9f81c39e7759 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:17:45.224Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-07-13T00:17:45.225Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.199.132:9042, scheduling retry in 2000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:17:45.612Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:17:45.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f3bab2e-8b85-4133-832a-863b8f6f864d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:17:45.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f3bab2e-8b85-4133-832a-863b8f6f864d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5a9c8aa0 2021-07-13T00:17:45.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f3bab2e-8b85-4133-832a-863b8f6f864d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:17:45.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f3bab2e-8b85-4133-832a-863b8f6f864d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:17:46.366Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=4f3bab2e-8b85-4133-832a-863b8f6f864d ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:17:46.366Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=4f3bab2e-8b85-4133-832a-863b8f6f864d ErrorCategory=ERROR ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:17:47.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf8518d0-c353-41af-a1ad-52cf1f3df671 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:17:47.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf8518d0-c353-41af-a1ad-52cf1f3df671 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@81d054a 2021-07-13T00:17:47.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf8518d0-c353-41af-a1ad-52cf1f3df671 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:17:47.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf8518d0-c353-41af-a1ad-52cf1f3df671 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:17:50.469Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=bf8518d0-c353-41af-a1ad-52cf1f3df671 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:17:50.470Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=bf8518d0-c353-41af-a1ad-52cf1f3df671 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:17:50.470Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=bf8518d0-c353-41af-a1ad-52cf1f3df671 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:17:50.471Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=bf8518d0-c353-41af-a1ad-52cf1f3df671 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:17:51.464Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fcc79fb8-72fa-49cb-8157-282a9fe55228 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:17:51.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fcc79fb8-72fa-49cb-8157-282a9fe55228 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@332218d2 2021-07-13T00:17:51.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fcc79fb8-72fa-49cb-8157-282a9fe55228 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:17:51.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fcc79fb8-72fa-49cb-8157-282a9fe55228 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:17:52.918Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:17:52.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:17:54.516Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=fcc79fb8-72fa-49cb-8157-282a9fe55228 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:17:54.517Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=fcc79fb8-72fa-49cb-8157-282a9fe55228 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:17:55.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dccea7da-8230-4018-9fae-ffccac400ff9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:17:55.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dccea7da-8230-4018-9fae-ffccac400ff9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63ff8d48 2021-07-13T00:17:55.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dccea7da-8230-4018-9fae-ffccac400ff9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:17:55.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dccea7da-8230-4018-9fae-ffccac400ff9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:17:58.500Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=dccea7da-8230-4018-9fae-ffccac400ff9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:17:58.500Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=dccea7da-8230-4018-9fae-ffccac400ff9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:17:59.464Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40b8ad15-5784-473c-9f9c-97460e10c160 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:17:59.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40b8ad15-5784-473c-9f9c-97460e10c160 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@206f9f17 2021-07-13T00:17:59.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40b8ad15-5784-473c-9f9c-97460e10c160 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:17:59.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40b8ad15-5784-473c-9f9c-97460e10c160 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:01.564Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:18:01.565Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:18:02.514Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=40b8ad15-5784-473c-9f9c-97460e10c160 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:18:02.515Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=40b8ad15-5784-473c-9f9c-97460e10c160 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:18:02.565Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:18:03.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41d02aaf-a1f6-4765-8b9f-e10d0b302a9e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:03.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41d02aaf-a1f6-4765-8b9f-e10d0b302a9e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@187b4750 2021-07-13T00:18:03.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41d02aaf-a1f6-4765-8b9f-e10d0b302a9e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:03.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41d02aaf-a1f6-4765-8b9f-e10d0b302a9e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:03.567Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:18:04.567Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:18:05.568Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:18:06.501Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=41d02aaf-a1f6-4765-8b9f-e10d0b302a9e ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:18:06.501Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=41d02aaf-a1f6-4765-8b9f-e10d0b302a9e ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:18:06.502Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=41d02aaf-a1f6-4765-8b9f-e10d0b302a9e ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:18:06.502Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=41d02aaf-a1f6-4765-8b9f-e10d0b302a9e ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:18:07.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:18:07.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=869ff368-f57a-459a-b519-6495082f8351 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:07.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=869ff368-f57a-459a-b519-6495082f8351 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@c5fd65 2021-07-13T00:18:07.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=869ff368-f57a-459a-b519-6495082f8351 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:07.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=869ff368-f57a-459a-b519-6495082f8351 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:18:07.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:18:08.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96b70783-64b7-4eb8-b154-12306d2e8a9c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:08.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96b70783-64b7-4eb8-b154-12306d2e8a9c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4dc1256e 2021-07-13T00:18:08.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96b70783-64b7-4eb8-b154-12306d2e8a9c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:08.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96b70783-64b7-4eb8-b154-12306d2e8a9c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:09.272Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=746331db-26ce-4f29-b82d-2fdaab2935c8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:09.272Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=746331db-26ce-4f29-b82d-2fdaab2935c8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6357e274 2021-07-13T00:18:09.272Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=746331db-26ce-4f29-b82d-2fdaab2935c8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:09.272Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=746331db-26ce-4f29-b82d-2fdaab2935c8 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:18:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:18:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:18:10.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:18:10.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdd285fa-f565-4624-90d0-5c76af25dcac PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:10.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdd285fa-f565-4624-90d0-5c76af25dcac PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4cd605ae 2021-07-13T00:18:10.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdd285fa-f565-4624-90d0-5c76af25dcac PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:10.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdd285fa-f565-4624-90d0-5c76af25dcac PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:11.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e955ae68-03e4-4911-83e6-0e05f6dff2db PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:11.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e955ae68-03e4-4911-83e6-0e05f6dff2db PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@569e740f 2021-07-13T00:18:11.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e955ae68-03e4-4911-83e6-0e05f6dff2db PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:11.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e955ae68-03e4-4911-83e6-0e05f6dff2db PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:12.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d223d13-3594-457d-addb-d5c655da292b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:12.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d223d13-3594-457d-addb-d5c655da292b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@12b0c1e7 2021-07-13T00:18:12.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d223d13-3594-457d-addb-d5c655da292b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:12.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d223d13-3594-457d-addb-d5c655da292b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:13.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:18:14.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa1c5b9d-e2a6-4e12-ab95-70dcc198ca2d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:14.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa1c5b9d-e2a6-4e12-ab95-70dcc198ca2d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ea8d587 2021-07-13T00:18:14.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa1c5b9d-e2a6-4e12-ab95-70dcc198ca2d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:14.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa1c5b9d-e2a6-4e12-ab95-70dcc198ca2d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:15.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=feb47af3-73ec-4f9c-ad03-4c149c484116 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:15.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=feb47af3-73ec-4f9c-ad03-4c149c484116 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@32af4289 2021-07-13T00:18:15.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=feb47af3-73ec-4f9c-ad03-4c149c484116 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:15.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=feb47af3-73ec-4f9c-ad03-4c149c484116 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:16.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e90adac-5f61-41c1-8973-59e8eff27943 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:16.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e90adac-5f61-41c1-8973-59e8eff27943 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@173476df 2021-07-13T00:18:16.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e90adac-5f61-41c1-8973-59e8eff27943 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:16.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8e90adac-5f61-41c1-8973-59e8eff27943 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:16.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:18:17.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5821eb9a-e566-4367-abc9-ef597f25e65a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:17.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5821eb9a-e566-4367-abc9-ef597f25e65a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@39f2227 2021-07-13T00:18:17.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5821eb9a-e566-4367-abc9-ef597f25e65a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:17.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5821eb9a-e566-4367-abc9-ef597f25e65a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:19.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d638f600-60ec-407d-a6e2-7d05cd8e240a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:19.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d638f600-60ec-407d-a6e2-7d05cd8e240a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@b13db3e 2021-07-13T00:18:19.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d638f600-60ec-407d-a6e2-7d05cd8e240a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:19.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d638f600-60ec-407d-a6e2-7d05cd8e240a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:19.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:18:20.263Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1e0cf65-a647-400d-8a65-f23e036c35d6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:20.263Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1e0cf65-a647-400d-8a65-f23e036c35d6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@43e06bdf 2021-07-13T00:18:20.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1e0cf65-a647-400d-8a65-f23e036c35d6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:20.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1e0cf65-a647-400d-8a65-f23e036c35d6 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:21.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c24582a1-34a8-4fab-ac63-fb29ed6bfa3c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:21.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c24582a1-34a8-4fab-ac63-fb29ed6bfa3c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@77566ff2 2021-07-13T00:18:21.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c24582a1-34a8-4fab-ac63-fb29ed6bfa3c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:21.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c24582a1-34a8-4fab-ac63-fb29ed6bfa3c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:22.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:18:22.464Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b4113235-60d0-4dd9-b0be-1d79a7484a44 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:22.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b4113235-60d0-4dd9-b0be-1d79a7484a44 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@da2f525 2021-07-13T00:18:22.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b4113235-60d0-4dd9-b0be-1d79a7484a44 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:22.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b4113235-60d0-4dd9-b0be-1d79a7484a44 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:22.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:18:22.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:18:23.657Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a380c510-9ccf-4448-bd34-f7c4dfafd055 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:23.657Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a380c510-9ccf-4448-bd34-f7c4dfafd055 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@18fd1584 2021-07-13T00:18:23.657Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a380c510-9ccf-4448-bd34-f7c4dfafd055 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:23.657Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a380c510-9ccf-4448-bd34-f7c4dfafd055 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:23.968Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.nfv.NsAntiAffinityRule to dao for create 2021-07-13T00:18:24.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f0bf67d-4681-413a-abea-87839258f41f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:24.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f0bf67d-4681-413a-abea-87839258f41f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@58032bd7 2021-07-13T00:18:24.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f0bf67d-4681-413a-abea-87839258f41f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:24.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f0bf67d-4681-413a-abea-87839258f41f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:25.460Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:18:26.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=902db38f-a36c-42dc-b577-3a22cce9d8a6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:26.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=902db38f-a36c-42dc-b577-3a22cce9d8a6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1ceee6da 2021-07-13T00:18:26.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=902db38f-a36c-42dc-b577-3a22cce9d8a6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:26.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=902db38f-a36c-42dc-b577-3a22cce9d8a6 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:27.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e168c6d-d347-42fe-9358-f69a1d207e5a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:27.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e168c6d-d347-42fe-9358-f69a1d207e5a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6ff33bf 2021-07-13T00:18:27.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e168c6d-d347-42fe-9358-f69a1d207e5a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:27.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e168c6d-d347-42fe-9358-f69a1d207e5a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:28.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca2b67d4-3e55-4bdf-977c-86918a744cb2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:28.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca2b67d4-3e55-4bdf-977c-86918a744cb2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@76de57d1 2021-07-13T00:18:28.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca2b67d4-3e55-4bdf-977c-86918a744cb2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:28.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca2b67d4-3e55-4bdf-977c-86918a744cb2 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:28.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:18:29.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc1dac7b-1161-424b-9a23-0b49aef7158a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:29.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc1dac7b-1161-424b-9a23-0b49aef7158a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3742974e 2021-07-13T00:18:29.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc1dac7b-1161-424b-9a23-0b49aef7158a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:29.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc1dac7b-1161-424b-9a23-0b49aef7158a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:29.710Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:18:29.874Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.nfv.Abstract.SecurityGroupRule to dao for create 2021-07-13T00:18:30.467Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:18:31.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4c4e3ec-3c2c-4dd9-b9b2-613d7548aa61 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:31.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4c4e3ec-3c2c-4dd9-b9b2-613d7548aa61 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5cf16f4a 2021-07-13T00:18:31.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4c4e3ec-3c2c-4dd9-b9b2-613d7548aa61 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:31.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c4c4e3ec-3c2c-4dd9-b9b2-613d7548aa61 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:31.271Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.nfv.NsSecurityGroupRule to dao for create 2021-07-13T00:18:32.260Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.nfv.NfpRule to dao for create 2021-07-13T00:18:32.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d5b53fb-d058-46bc-866d-c3f857e7a5f1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:32.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d5b53fb-d058-46bc-866d-c3f857e7a5f1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@236cdda0 2021-07-13T00:18:32.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d5b53fb-d058-46bc-866d-c3f857e7a5f1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:32.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d5b53fb-d058-46bc-866d-c3f857e7a5f1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:32.765Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6d5b53fb-d058-46bc-866d-c3f857e7a5f1 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:18:32.765Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6d5b53fb-d058-46bc-866d-c3f857e7a5f1 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:18:33.470Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.nfv.NsMonitoring to dao for create 2021-07-13T00:18:34.270Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=758a3ae3-b81b-427f-8c63-7f00d9a9aa0a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:34.270Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=758a3ae3-b81b-427f-8c63-7f00d9a9aa0a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@278866c3 2021-07-13T00:18:34.270Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=758a3ae3-b81b-427f-8c63-7f00d9a9aa0a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:34.270Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=758a3ae3-b81b-427f-8c63-7f00d9a9aa0a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:34.559Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp send POLICY_TYPE : tosca.policies.nfv.VnfMonitoring to dao for create 2021-07-13T00:18:34.576Z [qtp817348612-16] INFO o.o.s.b.c.impl.CommonImportManager RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp all PolicyTypeDataDefinition [uniqueId=tosca.policies.nfv.VnfMonitoring.1.0.policytype, type=tosca.policies.nfv.VnfMonitoring, name=NFV VnfMonitoring, icon=policy-nfv-anti-affinityrule, version=1.0, targets=[tosca.nodes.nfv.VNF], metadata=null, description=Policy type is used to identify information to be monitored during the lifetime of a VNF instance as defined in ETSI GS NFV-IFA 014 [2]., creationTime=1626135514563, modificationTime=1626135514563] [ properties=[PropertyDefinition [ PropertyDataDefinition(definition=false, hidden=false, uniqueId=tosca.policies.nfv.VnfMonitoring.1.0.policytype.vnf_monitoring_parameters, type=list, required=true, defaultValue=null, description=Specifies a virtualised resource related performance metric to be monitored on the NS level., schema=SchemaDefinition [derivedFrom='null, constraints=null, properties=null, property=PropertyDefinition [ PropertyDataDefinition(definition=false, hidden=false, uniqueId=null, type=tosca.datatypes.nfv.VnfMonitoringParameter, required=false, defaultValue=null, description=null, schema=null, password=false, name=null, value=null, label=null, immutable=false, mappedToComponentProperty=true, toscaGetFunctionType=null, inputPath=null, status=null, inputId=null, instanceUniqueId=null, propertyId=null, parentPropertyType=null, subPropertyInputPath=null, annotations=null, parentUniqueId=null, getInputValues=null, isDeclaredListInput=false, getPolicyValues=null, propertyConstraints=null, metadata=null), name=null, constraints=null]]], password=false, name=vnf_monitoring_parameters, value=null, label=null, immutable=false, mappedToComponentProperty=true, toscaGetFunctionType=null, inputPath=null, status=null, inputId=null, instanceUniqueId=null, propertyId=null, parentPropertyType=null, subPropertyInputPath=null, annotations=null, parentUniqueId=null, getInputValues=null, isDeclaredListInput=false, getPolicyValues=null, propertyConstraints=[{"minLength":1}], metadata=null), name=vnf_monitoring_parameters, constraints=[org.openecomp.sdc.be.model.tosca.constraints.MinLengthConstraint@2867d0cb]]]] ] were created successfully!!! 2021-07-13T00:18:35.362Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fcbcdead-fa42-4c85-a1a5-ccc81b1ac885 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:35.362Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fcbcdead-fa42-4c85-a1a5-ccc81b1ac885 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4d36d9b 2021-07-13T00:18:35.362Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fcbcdead-fa42-4c85-a1a5-ccc81b1ac885 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:35.362Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fcbcdead-fa42-4c85-a1a5-ccc81b1ac885 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:36.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=006fc3ce-22bc-47e4-864a-b8a7a4ef1892 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:36.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=006fc3ce-22bc-47e4-864a-b8a7a4ef1892 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@530fd974 2021-07-13T00:18:36.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=006fc3ce-22bc-47e4-864a-b8a7a4ef1892 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:36.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=006fc3ce-22bc-47e4-864a-b8a7a4ef1892 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:37.379Z [qtp817348612-16] WARN o.o.s.be.servlets.TypesUploadServlet RequestID=null ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp #uploadPolicyTypes('{"tosca.policies.Root": {"displayName": "Root", "icon": "policy-root"}, "tosca.policies.Placement": {"displayName": "Placement", "icon": "policy-placement"}, "tosca.policies.Scaling": {"displayName": "Scaling", "icon": "policy-scaling"}, "tosca.policies.Update": {"displayName": "Update", "icon": "policy-update"}, "tosca.policies.Performance": {"displayName": "Performance", "icon": "policy-performance"}, "org.openecomp.policies.placement.Antilocate": {"displayName": "Placement Antilocate", "icon": "policy-placement-antilocate"}, "org.openecomp.policies.placement.Colocate": {"displayName": "Placement Colocate", "icon": "policy-placement-colocate"}, "org.openecomp.policies.placement.valet.Diversity": {"displayName": "Placement Valet Diversity", "icon": "policy-placement-valet-diversity"}, "org.openecomp.policies.placement.valet.Exclusivity": {"displayName": "Placement Valet Exclusivity", "icon": "policy-placement-valet-exclusivity"}, "org.openecomp.policies.placement.valet.Affinity": {"displayName": "Placement Valet Affinity", "icon": "placement-valet-affinity"}, "org.openecomp.policies.scaling.Fixed": {"displayName": "Scaling Fixed", "icon": "policy-fixed"}, "org.openecomp.policies.External": {"displayName": "External", "icon": "policy-external"}, "tosca.policies.nfv.InstantiationLevels": {"displayName": "NFV InstantiationLevels", "icon": "policy-nfv-instantiationlevels"}, "tosca.policies.nfv.VduInstantiationLevels": {"displayName": "NFV Vdu InstantiationLevels", "icon": "policy-nfv-vdu-instantiationlevels"}, "tosca.policies.nfv.VirtualLinkInstantiationLevels": {"displayName": "NFV VirtualLink InstantiationLevels", "icon": "policy-nfv-vl-instantiationlevels"}, "tosca.policies.nfv.ScalingAspects": {"displayName": "NFV ScalingAspects", "icon": "policy-nfv-scalingaspects"}, "tosca.policies.nfv.VduScalingAspectDeltas": {"displayName": "NFV Vdu ScalingAspectDeltas", "icon": "policy-nfv-vdu-scalingAspectdeltas"}, "tosca.policies.nfv.VirtualLinkBitrateScalingAspectDeltas": {"displayName": "NFV VirtualLinkBitrate ScalingAspectDeltas", "icon": "policy-nfv-vlbitrate-scalingAspectdeltas"}, "tosca.policies.nfv.VduInitialDelta": {"displayName": "NFV Vdu InitialDelta", "icon": "policy-nfv-vdu-initialdelta"}, "tosca.policies.nfv.VirtualLinkBitrateInitialDelta": {"displayName": "NFV VirtualLinkBitrate InitialDelta", "icon": "policy-nfv-vlbitrate-initialdelta"}, "tosca.policies.nfv.SecurityGroupRule": {"displayName": "NFV SecurityGroupRule", "icon": "policy-nfv-securitygrouprule"}, "tosca.policies.nfv.SupportedVnfInterface": {"displayName": "NFV Supported VnfInterface", "icon": "policy-nfv-supported-vnfinterface"}, "tosca.policies.nfv.AffinityRule": {"displayName": "NFV AffinityRule", "icon": "policy-nfv-affinityrule"}, "tosca.policies.nfv.AntiAffinityRule": {"displayName": "NFV AntiAffinityRule", "icon": "policy-nfv-anti-affinityrule"}, "tosca.policies.nfv.NsAffinityRule": {"displayName": "NFV NsAffinityRule", "icon": "policy-nfv-anti-affinityrule"}, "tosca.policies.nfv.NsAntiAffinityRule": {"displayName": "NFV NsAntiAffinityRule", "icon": "policy-nfv-anti-affinityrule"}, "tosca.policies.nfv.Abstract.SecurityGroupRule": {"displayName": "NFV Abstract SecurityGroupRule", "icon": "policy-nfv-anti-affinityrule"}, "tosca.policies.nfv.NsSecurityGroupRule": {"displayName": "NFV NsSecurityGroupRule", "icon": "policy-nfv-anti-affinityrule"}, "tosca.policies.nfv.NfpRule": {"displayName": "NFV NfpRule ", "icon": "policy-nfv-anti-affinityrule"}, "tosca.policies.nfv.NsMonitoring": {"displayName": "NFV NsMonitoring", "icon": "policy-nfv-anti-affinityrule"}, "tosca.policies.nfv.VnfMonitoring": {"displayName": "NFV VnfMonitoring", "icon": "policy-nfv-anti-affinityrule"}}', /tmp/rep2538114483533197556tmp, 'Request(POST https://sdc-be.onap:8443/sdc2/rest/v1/catalog/uploadType/policytypes)@62228854', 'jh0003'): org.apache.cxf.jaxrs.impl.ResponseImpl@75e31480 in 5.23min (too slow!) 2021-07-13T00:18:37.382Z [qtp817348612-16] INFO o.o.l.f.base.AbstractAuditLogFilter ResponseCode=201 RequestID=null ElapsedTime=313815 PartnerName=jh0003 auditOn=true ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp StatusCode=COMPLETE ServerIPAddress=10.242.98.212 ServiceName=/v1/catalog/uploadType/policytypes EntryTimestamp=2021-07-13T00:13:23.566Z InvocationID=c420bd0d-f642-439e-a3ae-6ce2832e5326 LogTimestamp=2021-07-13T00:18:37.381197Z ResponseDesc=Created Exiting. 2021-07-13T00:18:37.382Z [qtp817348612-16] INFO o.o.sdc.be.filters.BeServletFilter RequestID=null POST /sdc2/rest/v1/catalog/uploadType/policytypes HTTP/1.1 SC="201" 2021-07-13T00:18:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:18:37.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:18:38.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c756637d-fe6a-4ecf-9f67-ed23d2520163 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:38.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c756637d-fe6a-4ecf-9f67-ed23d2520163 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@16cb8f68 2021-07-13T00:18:38.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c756637d-fe6a-4ecf-9f67-ed23d2520163 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:38.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c756637d-fe6a-4ecf-9f67-ed23d2520163 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:39.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af91dcd1-d18d-4ac9-b0cd-770bac92973d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:39.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af91dcd1-d18d-4ac9-b0cd-770bac92973d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@72914d0c 2021-07-13T00:18:39.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af91dcd1-d18d-4ac9-b0cd-770bac92973d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:39.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af91dcd1-d18d-4ac9-b0cd-770bac92973d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:40.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90e3a7aa-5877-4e12-a7a9-788a21c3b72b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:40.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90e3a7aa-5877-4e12-a7a9-788a21c3b72b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6db15970 2021-07-13T00:18:40.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90e3a7aa-5877-4e12-a7a9-788a21c3b72b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:40.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90e3a7aa-5877-4e12-a7a9-788a21c3b72b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:42.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=68e3f27d-a439-40ea-818d-fad548d17bb7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:42.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=68e3f27d-a439-40ea-818d-fad548d17bb7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4058e9cb 2021-07-13T00:18:42.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=68e3f27d-a439-40ea-818d-fad548d17bb7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:42.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=68e3f27d-a439-40ea-818d-fad548d17bb7 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:43.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10b137a0-9157-4935-8348-1fa44b60808a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:43.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10b137a0-9157-4935-8348-1fa44b60808a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@aa29e67 2021-07-13T00:18:43.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10b137a0-9157-4935-8348-1fa44b60808a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:43.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10b137a0-9157-4935-8348-1fa44b60808a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:43.060Z [BE-Health-Check-Task] INFO o.a.http.impl.execchain.RetryExec RequestID=10b137a0-9157-4935-8348-1fa44b60808a PartnerName=SDC.BE I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {s}->https://sdc-onboarding-be.onap:8445: The target server failed to respond 2021-07-13T00:18:43.060Z [BE-Health-Check-Task] INFO o.a.http.impl.execchain.RetryExec RequestID=10b137a0-9157-4935-8348-1fa44b60808a PartnerName=SDC.BE Retrying request to {s}->https://sdc-onboarding-be.onap:8445 2021-07-13T00:18:43.064Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=10b137a0-9157-4935-8348-1fa44b60808a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) 2021-07-13T00:18:43.064Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=10b137a0-9157-4935-8348-1fa44b60808a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:18:43.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=483e6bbf-cdde-43ee-88a6-b6d262ac247c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:43.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=483e6bbf-cdde-43ee-88a6-b6d262ac247c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@15bee6ff 2021-07-13T00:18:43.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=483e6bbf-cdde-43ee-88a6-b6d262ac247c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:43.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=483e6bbf-cdde-43ee-88a6-b6d262ac247c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:43.962Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=483e6bbf-cdde-43ee-88a6-b6d262ac247c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) 2021-07-13T00:18:43.963Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=483e6bbf-cdde-43ee-88a6-b6d262ac247c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:18:45.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=22204bbc-85cf-4f83-9242-8b114d248929 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:45.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=22204bbc-85cf-4f83-9242-8b114d248929 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@c60a6b6 2021-07-13T00:18:45.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=22204bbc-85cf-4f83-9242-8b114d248929 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:45.166Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=22204bbc-85cf-4f83-9242-8b114d248929 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:45.168Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=22204bbc-85cf-4f83-9242-8b114d248929 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) 2021-07-13T00:18:45.168Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=22204bbc-85cf-4f83-9242-8b114d248929 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:18:45.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47c57726-f9c7-4eee-bdc7-0aa66bb53d77 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:45.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47c57726-f9c7-4eee-bdc7-0aa66bb53d77 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6989c9c 2021-07-13T00:18:45.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47c57726-f9c7-4eee-bdc7-0aa66bb53d77 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:45.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47c57726-f9c7-4eee-bdc7-0aa66bb53d77 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:45.771Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=47c57726-f9c7-4eee-bdc7-0aa66bb53d77 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) 2021-07-13T00:18:45.771Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=47c57726-f9c7-4eee-bdc7-0aa66bb53d77 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:18:46.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=749645af-a8c5-47f1-bcda-123ae25644a2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:46.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=749645af-a8c5-47f1-bcda-123ae25644a2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@671aa7d6 2021-07-13T00:18:46.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=749645af-a8c5-47f1-bcda-123ae25644a2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:46.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=749645af-a8c5-47f1-bcda-123ae25644a2 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:46.665Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=749645af-a8c5-47f1-bcda-123ae25644a2 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) 2021-07-13T00:18:46.665Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=749645af-a8c5-47f1-bcda-123ae25644a2 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:18:47.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49c56dcd-a737-4539-90e8-a321fc771cad PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:47.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49c56dcd-a737-4539-90e8-a321fc771cad PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@59146bcc 2021-07-13T00:18:47.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49c56dcd-a737-4539-90e8-a321fc771cad PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:47.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49c56dcd-a737-4539-90e8-a321fc771cad PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:47.565Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=49c56dcd-a737-4539-90e8-a321fc771cad ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) 2021-07-13T00:18:47.565Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=49c56dcd-a737-4539-90e8-a321fc771cad ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:18:48.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=872ea21d-068d-4f4b-8b4b-492850a3c0a3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:48.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=872ea21d-068d-4f4b-8b4b-492850a3c0a3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@32d04180 2021-07-13T00:18:48.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=872ea21d-068d-4f4b-8b4b-492850a3c0a3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:48.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=872ea21d-068d-4f4b-8b4b-492850a3c0a3 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:48.776Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=872ea21d-068d-4f4b-8b4b-492850a3c0a3 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) 2021-07-13T00:18:48.777Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=872ea21d-068d-4f4b-8b4b-492850a3c0a3 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.base/java.net.SocksSocketImpl.connect(Unknown Source) at java.base/java.net.Socket.connect(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:18:49.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81009967-dd96-41ed-9668-0b3615d30e8d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:49.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81009967-dd96-41ed-9668-0b3615d30e8d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@c0c1ba 2021-07-13T00:18:49.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81009967-dd96-41ed-9668-0b3615d30e8d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:49.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81009967-dd96-41ed-9668-0b3615d30e8d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:52.872Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=81009967-dd96-41ed-9668-0b3615d30e8d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out 2021-07-13T00:18:52.872Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=81009967-dd96-41ed-9668-0b3615d30e8d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:18:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:18:52.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:18:54.360Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=579e3322-7155-48ad-860a-eb30f98a0643 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:54.360Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=579e3322-7155-48ad-860a-eb30f98a0643 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3cbb474 2021-07-13T00:18:54.360Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=579e3322-7155-48ad-860a-eb30f98a0643 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:54.360Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=579e3322-7155-48ad-860a-eb30f98a0643 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:18:57.378Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=579e3322-7155-48ad-860a-eb30f98a0643 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out 2021-07-13T00:18:57.378Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=579e3322-7155-48ad-860a-eb30f98a0643 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:18:58.472Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e226ed10-b62e-45a7-ba53-fe1f5ed1b8f9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:18:58.472Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e226ed10-b62e-45a7-ba53-fe1f5ed1b8f9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7b776925 2021-07-13T00:18:58.472Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e226ed10-b62e-45a7-ba53-fe1f5ed1b8f9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:18:58.472Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e226ed10-b62e-45a7-ba53-fe1f5ed1b8f9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:01.492Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=e226ed10-b62e-45a7-ba53-fe1f5ed1b8f9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out 2021-07-13T00:19:01.493Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=e226ed10-b62e-45a7-ba53-fe1f5ed1b8f9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:19:02.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18b41480-6395-47cc-ab68-3fbf3b44e46e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:02.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18b41480-6395-47cc-ab68-3fbf3b44e46e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2f926a69 2021-07-13T00:19:02.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18b41480-6395-47cc-ab68-3fbf3b44e46e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:02.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18b41480-6395-47cc-ab68-3fbf3b44e46e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:05.272Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=18b41480-6395-47cc-ab68-3fbf3b44e46e ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out 2021-07-13T00:19:05.273Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=18b41480-6395-47cc-ab68-3fbf3b44e46e ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:19:06.263Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89c194a4-c222-4917-a622-ad3838b231da PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:06.263Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89c194a4-c222-4917-a622-ad3838b231da PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@779919d8 2021-07-13T00:19:06.263Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89c194a4-c222-4917-a622-ad3838b231da PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:06.263Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89c194a4-c222-4917-a622-ad3838b231da PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:19:07.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:19:09.279Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=89c194a4-c222-4917-a622-ad3838b231da ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out 2021-07-13T00:19:09.280Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=89c194a4-c222-4917-a622-ad3838b231da ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:19:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:19:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:19:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:19:10.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50c9a378-7774-4e79-989e-18fd3b8a9caf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:10.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50c9a378-7774-4e79-989e-18fd3b8a9caf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1ad065f8 2021-07-13T00:19:10.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50c9a378-7774-4e79-989e-18fd3b8a9caf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:10.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50c9a378-7774-4e79-989e-18fd3b8a9caf PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:13.978Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=50c9a378-7774-4e79-989e-18fd3b8a9caf ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out 2021-07-13T00:19:13.979Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=50c9a378-7774-4e79-989e-18fd3b8a9caf ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:19:14.784Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3343098-dd2d-4123-a5e7-df1c537fb488 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:14.784Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3343098-dd2d-4123-a5e7-df1c537fb488 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@29d16f44 2021-07-13T00:19:14.784Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3343098-dd2d-4123-a5e7-df1c537fb488 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:14.784Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3343098-dd2d-4123-a5e7-df1c537fb488 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:17.801Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=f3343098-dd2d-4123-a5e7-df1c537fb488 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out 2021-07-13T00:19:17.802Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=f3343098-dd2d-4123-a5e7-df1c537fb488 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:19:18.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a726d3d-3c61-4960-bdf1-8abacff97580 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:18.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a726d3d-3c61-4960-bdf1-8abacff97580 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7ac33013 2021-07-13T00:19:18.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a726d3d-3c61-4960-bdf1-8abacff97580 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:18.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a726d3d-3c61-4960-bdf1-8abacff97580 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:21.875Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=5a726d3d-3c61-4960-bdf1-8abacff97580 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out 2021-07-13T00:19:21.876Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=5a726d3d-3c61-4960-bdf1-8abacff97580 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T00:19:22.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=879ee7d2-14d0-4b2d-b2f2-4cdc516cb5fa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:22.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=879ee7d2-14d0-4b2d-b2f2-4cdc516cb5fa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b61956 2021-07-13T00:19:22.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=879ee7d2-14d0-4b2d-b2f2-4cdc516cb5fa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:22.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=879ee7d2-14d0-4b2d-b2f2-4cdc516cb5fa PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:22.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:19:22.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:19:24.416Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:19:25.921Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=879ee7d2-14d0-4b2d-b2f2-4cdc516cb5fa ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:19:25.921Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=879ee7d2-14d0-4b2d-b2f2-4cdc516cb5fa ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:19:26.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42c50141-21f3-4f4e-b343-adbd455df609 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:26.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42c50141-21f3-4f4e-b343-adbd455df609 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2a4a596a 2021-07-13T00:19:26.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42c50141-21f3-4f4e-b343-adbd455df609 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:26.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42c50141-21f3-4f4e-b343-adbd455df609 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:30.030Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=42c50141-21f3-4f4e-b343-adbd455df609 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:19:30.030Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=42c50141-21f3-4f4e-b343-adbd455df609 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:19:31.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b13f043a-8268-4a6b-af64-240fbd720d8f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:31.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b13f043a-8268-4a6b-af64-240fbd720d8f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@472dc813 2021-07-13T00:19:31.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b13f043a-8268-4a6b-af64-240fbd720d8f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:31.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b13f043a-8268-4a6b-af64-240fbd720d8f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:34.121Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=b13f043a-8268-4a6b-af64-240fbd720d8f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:19:34.121Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=b13f043a-8268-4a6b-af64-240fbd720d8f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:19:34.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=43171a57-2160-4cca-bd5c-050759357607 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:34.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=43171a57-2160-4cca-bd5c-050759357607 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@237dad42 2021-07-13T00:19:34.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=43171a57-2160-4cca-bd5c-050759357607 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:34.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=43171a57-2160-4cca-bd5c-050759357607 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:37.917Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:19:37.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:19:38.568Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15e54163-fc87-41b8-92de-8b3fff1b4b6b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:38.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15e54163-fc87-41b8-92de-8b3fff1b4b6b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7cb674e3 2021-07-13T00:19:38.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15e54163-fc87-41b8-92de-8b3fff1b4b6b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:38.571Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15e54163-fc87-41b8-92de-8b3fff1b4b6b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:40.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fd5f709-a44e-4632-bb67-510ccec9b62e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:40.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fd5f709-a44e-4632-bb67-510ccec9b62e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63a2b82 2021-07-13T00:19:40.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fd5f709-a44e-4632-bb67-510ccec9b62e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:40.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fd5f709-a44e-4632-bb67-510ccec9b62e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:41.470Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=959a98c0-65f7-4426-bd84-22a9629e4eec PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:41.471Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=959a98c0-65f7-4426-bd84-22a9629e4eec PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@48e381ba 2021-07-13T00:19:41.471Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=959a98c0-65f7-4426-bd84-22a9629e4eec PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:41.471Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=959a98c0-65f7-4426-bd84-22a9629e4eec PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:42.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31359c3e-64ff-470c-91b0-de2a1d37507c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:42.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31359c3e-64ff-470c-91b0-de2a1d37507c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@fcec10b 2021-07-13T00:19:42.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31359c3e-64ff-470c-91b0-de2a1d37507c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:42.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31359c3e-64ff-470c-91b0-de2a1d37507c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:44.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3129a86-cbf6-4079-babb-d30c016b5c78 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:44.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3129a86-cbf6-4079-babb-d30c016b5c78 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4010bd7d 2021-07-13T00:19:44.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3129a86-cbf6-4079-babb-d30c016b5c78 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:44.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3129a86-cbf6-4079-babb-d30c016b5c78 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:45.456Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a904fe4-659b-47a8-8b63-e2d0b845de70 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:45.456Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a904fe4-659b-47a8-8b63-e2d0b845de70 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@17a1f8c6 2021-07-13T00:19:45.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a904fe4-659b-47a8-8b63-e2d0b845de70 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:45.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a904fe4-659b-47a8-8b63-e2d0b845de70 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:46.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f897b87d-5e4a-43c4-b500-7dc5a85996d1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:46.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f897b87d-5e4a-43c4-b500-7dc5a85996d1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@439b45d9 2021-07-13T00:19:46.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f897b87d-5e4a-43c4-b500-7dc5a85996d1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:46.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f897b87d-5e4a-43c4-b500-7dc5a85996d1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:48.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92d8cbb2-4ddb-49c6-b364-bb0457b3ecc2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:48.259Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92d8cbb2-4ddb-49c6-b364-bb0457b3ecc2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3a240238 2021-07-13T00:19:48.259Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92d8cbb2-4ddb-49c6-b364-bb0457b3ecc2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:48.259Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92d8cbb2-4ddb-49c6-b364-bb0457b3ecc2 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:49.166Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=530d23d7-d5ca-49ad-bb2d-a46656186248 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:49.167Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=530d23d7-d5ca-49ad-bb2d-a46656186248 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f9abb54 2021-07-13T00:19:49.167Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=530d23d7-d5ca-49ad-bb2d-a46656186248 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:49.167Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=530d23d7-d5ca-49ad-bb2d-a46656186248 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:50.367Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a835e0c1-4614-4504-b627-5740c3ed7379 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:50.367Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a835e0c1-4614-4504-b627-5740c3ed7379 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@67e50ba9 2021-07-13T00:19:50.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a835e0c1-4614-4504-b627-5740c3ed7379 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:50.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a835e0c1-4614-4504-b627-5740c3ed7379 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:51.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e22ec903-6765-45ec-b374-f233d6f3d8d3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:51.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e22ec903-6765-45ec-b374-f233d6f3d8d3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7e492f01 2021-07-13T00:19:51.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e22ec903-6765-45ec-b374-f233d6f3d8d3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:51.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e22ec903-6765-45ec-b374-f233d6f3d8d3 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:19:52.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:19:52.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8bc6bb43-3c56-4da5-b316-df9e06b35cc2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:52.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8bc6bb43-3c56-4da5-b316-df9e06b35cc2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64e6c732 2021-07-13T00:19:52.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8bc6bb43-3c56-4da5-b316-df9e06b35cc2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:52.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8bc6bb43-3c56-4da5-b316-df9e06b35cc2 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:54.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d836c583-e297-4483-9e35-57629400769b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:54.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d836c583-e297-4483-9e35-57629400769b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4e8c90e1 2021-07-13T00:19:54.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d836c583-e297-4483-9e35-57629400769b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:54.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d836c583-e297-4483-9e35-57629400769b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:55.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=80fa79be-ac8d-4e4b-813b-25efac055472 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:55.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=80fa79be-ac8d-4e4b-813b-25efac055472 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1fb62da 2021-07-13T00:19:55.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=80fa79be-ac8d-4e4b-813b-25efac055472 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:55.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=80fa79be-ac8d-4e4b-813b-25efac055472 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:56.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8ae50ac-4752-43a7-ae94-f0a52e983638 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:56.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8ae50ac-4752-43a7-ae94-f0a52e983638 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@367d64c3 2021-07-13T00:19:56.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8ae50ac-4752-43a7-ae94-f0a52e983638 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:56.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8ae50ac-4752-43a7-ae94-f0a52e983638 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:57.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e4cb5ac-c0b7-4ff2-a141-0998f54f491f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:57.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e4cb5ac-c0b7-4ff2-a141-0998f54f491f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64f0ea69 2021-07-13T00:19:57.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e4cb5ac-c0b7-4ff2-a141-0998f54f491f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:57.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2e4cb5ac-c0b7-4ff2-a141-0998f54f491f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:19:58.538Z [JanusGraph Cluster-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host /10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. 2021-07-13T00:19:58.540Z [JanusGraph Cluster-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to /10.242.199.132:9042, scheduling retry in 32000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:19:59.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8109343-fd10-4f03-a4e1-bbad413008b6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:19:59.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8109343-fd10-4f03-a4e1-bbad413008b6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ed1758e 2021-07-13T00:19:59.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8109343-fd10-4f03-a4e1-bbad413008b6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:19:59.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8109343-fd10-4f03-a4e1-bbad413008b6 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:00.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c19dcd39-76ec-480a-836f-7beb496aab9b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:00.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c19dcd39-76ec-480a-836f-7beb496aab9b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1466233c 2021-07-13T00:20:00.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c19dcd39-76ec-480a-836f-7beb496aab9b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:00.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c19dcd39-76ec-480a-836f-7beb496aab9b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:00.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a062f7bd-68a4-4235-a796-c84f963f62a0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:00.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a062f7bd-68a4-4235-a796-c84f963f62a0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@49e0060f 2021-07-13T00:20:00.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a062f7bd-68a4-4235-a796-c84f963f62a0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:00.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a062f7bd-68a4-4235-a796-c84f963f62a0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:01.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a66e1dd-0ff6-4e13-a57b-2ed3c02b7495 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:01.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a66e1dd-0ff6-4e13-a57b-2ed3c02b7495 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2aa01ef8 2021-07-13T00:20:01.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a66e1dd-0ff6-4e13-a57b-2ed3c02b7495 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:01.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a66e1dd-0ff6-4e13-a57b-2ed3c02b7495 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:03.074Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b4312b1-25be-4078-8f88-6b6f8f51724d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:03.074Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b4312b1-25be-4078-8f88-6b6f8f51724d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6f186327 2021-07-13T00:20:03.074Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b4312b1-25be-4078-8f88-6b6f8f51724d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:03.074Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b4312b1-25be-4078-8f88-6b6f8f51724d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:04.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91f37fbb-94f7-4902-8f71-1ea3d75a7bc3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:04.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91f37fbb-94f7-4902-8f71-1ea3d75a7bc3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4a42c23b 2021-07-13T00:20:04.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91f37fbb-94f7-4902-8f71-1ea3d75a7bc3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:04.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91f37fbb-94f7-4902-8f71-1ea3d75a7bc3 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:05.273Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ec4ec63-bbd6-4ba4-9117-af3cc02c4e07 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:05.273Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ec4ec63-bbd6-4ba4-9117-af3cc02c4e07 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@35b7502 2021-07-13T00:20:05.273Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ec4ec63-bbd6-4ba4-9117-af3cc02c4e07 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:05.273Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ec4ec63-bbd6-4ba4-9117-af3cc02c4e07 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:06.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5308b36-ee21-4870-a7d4-f0aa9d10be75 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:06.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5308b36-ee21-4870-a7d4-f0aa9d10be75 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2926fc4f 2021-07-13T00:20:06.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5308b36-ee21-4870-a7d4-f0aa9d10be75 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:06.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5308b36-ee21-4870-a7d4-f0aa9d10be75 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:07.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11e295d7-5277-432f-baf9-939ba145f5f7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:07.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11e295d7-5277-432f-baf9-939ba145f5f7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@50ba317 2021-07-13T00:20:07.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11e295d7-5277-432f-baf9-939ba145f5f7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:07.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11e295d7-5277-432f-baf9-939ba145f5f7 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:20:07.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:20:08.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=319eb1be-688e-4b76-aa73-ee85de22e983 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:08.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=319eb1be-688e-4b76-aa73-ee85de22e983 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5984ea97 2021-07-13T00:20:08.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=319eb1be-688e-4b76-aa73-ee85de22e983 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:08.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=319eb1be-688e-4b76-aa73-ee85de22e983 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:20:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:20:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:20:09.723Z [cluster2-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-07-13T00:20:09.724Z [cluster2-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.199.132:9042, scheduling retry in 30000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:20:10.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17a8ddc8-4922-41fc-a02c-07c7ae90c288 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:10.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17a8ddc8-4922-41fc-a02c-07c7ae90c288 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1af67c0 2021-07-13T00:20:10.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17a8ddc8-4922-41fc-a02c-07c7ae90c288 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:10.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=17a8ddc8-4922-41fc-a02c-07c7ae90c288 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:10.636Z [cluster1-reconnection-1] ERROR c.d.d.c.AbstractReconnectionHandler Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE 2021-07-13T00:20:10.637Z [cluster1-reconnection-1] ERROR com.datastax.driver.core.Cluster Authentication error during reconnection to cassandra.onap/10.242.199.132:9042, scheduling retry in 64000 milliseconds com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.199.132:9042: Error during authentication of user asdc_user : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:20:11.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ecd5713-af22-40f3-ae18-424fa3ddb218 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:11.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ecd5713-af22-40f3-ae18-424fa3ddb218 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f75273e 2021-07-13T00:20:11.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ecd5713-af22-40f3-ae18-424fa3ddb218 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:11.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ecd5713-af22-40f3-ae18-424fa3ddb218 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:12.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b0964e37-f091-4ca7-95a5-aa9bc5530e89 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:12.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b0964e37-f091-4ca7-95a5-aa9bc5530e89 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@473e2cdc 2021-07-13T00:20:12.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b0964e37-f091-4ca7-95a5-aa9bc5530e89 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:12.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b0964e37-f091-4ca7-95a5-aa9bc5530e89 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:13.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49b9b9ab-99b5-4ce6-b59e-732138e417e1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:13.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49b9b9ab-99b5-4ce6-b59e-732138e417e1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3910a6fe 2021-07-13T00:20:13.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49b9b9ab-99b5-4ce6-b59e-732138e417e1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:13.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49b9b9ab-99b5-4ce6-b59e-732138e417e1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:15.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8585330b-ae8c-4542-859c-c4363d0f1fc9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:15.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8585330b-ae8c-4542-859c-c4363d0f1fc9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@43fb477b 2021-07-13T00:20:15.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8585330b-ae8c-4542-859c-c4363d0f1fc9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:15.157Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8585330b-ae8c-4542-859c-c4363d0f1fc9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:15.573Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=8585330b-ae8c-4542-859c-c4363d0f1fc9 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:20:15.573Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=8585330b-ae8c-4542-859c-c4363d0f1fc9 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:20:16.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=29f93eed-ec6a-4c3f-a2d8-09c33985431b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:16.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=29f93eed-ec6a-4c3f-a2d8-09c33985431b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7501c4ed 2021-07-13T00:20:16.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=29f93eed-ec6a-4c3f-a2d8-09c33985431b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:16.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=29f93eed-ec6a-4c3f-a2d8-09c33985431b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:18.468Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7a83c2e-6a3d-4f40-801c-720d18a52e37 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:18.468Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7a83c2e-6a3d-4f40-801c-720d18a52e37 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@43ef5e5d 2021-07-13T00:20:18.468Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7a83c2e-6a3d-4f40-801c-720d18a52e37 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:18.468Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7a83c2e-6a3d-4f40-801c-720d18a52e37 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:19.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab8da3d6-080e-460d-8c22-671b487bd73c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:19.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab8da3d6-080e-460d-8c22-671b487bd73c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7ff43788 2021-07-13T00:20:19.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab8da3d6-080e-460d-8c22-671b487bd73c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:19.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab8da3d6-080e-460d-8c22-671b487bd73c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:21.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=082d1fc7-d359-4742-b62e-e641aa7339e6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:21.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=082d1fc7-d359-4742-b62e-e641aa7339e6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@662d7792 2021-07-13T00:20:21.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=082d1fc7-d359-4742-b62e-e641aa7339e6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:21.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=082d1fc7-d359-4742-b62e-e641aa7339e6 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:20:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:20:23.367Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aaa6e9e2-a517-45d1-b056-badd577a8673 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:23.367Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aaa6e9e2-a517-45d1-b056-badd577a8673 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1d70e116 2021-07-13T00:20:23.367Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aaa6e9e2-a517-45d1-b056-badd577a8673 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:23.367Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aaa6e9e2-a517-45d1-b056-badd577a8673 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:24.723Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:20:25.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e956b99-e2f5-4d34-b264-94214fb320ff PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:25.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e956b99-e2f5-4d34-b264-94214fb320ff PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@24d45953 2021-07-13T00:20:25.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e956b99-e2f5-4d34-b264-94214fb320ff PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:25.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e956b99-e2f5-4d34-b264-94214fb320ff PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:27.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa75e441-87c7-45fa-b261-3332d5c27874 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:27.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa75e441-87c7-45fa-b261-3332d5c27874 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7ac26236 2021-07-13T00:20:27.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa75e441-87c7-45fa-b261-3332d5c27874 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:27.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa75e441-87c7-45fa-b261-3332d5c27874 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:29.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ff9d77c-a098-4298-a980-1815dac28f61 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:29.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ff9d77c-a098-4298-a980-1815dac28f61 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@77de1b1b 2021-07-13T00:20:29.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ff9d77c-a098-4298-a980-1815dac28f61 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:29.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ff9d77c-a098-4298-a980-1815dac28f61 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:32.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=754da02d-b699-4a8a-95ff-f4ad3bf7d788 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:32.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=754da02d-b699-4a8a-95ff-f4ad3bf7d788 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6dbc2ca7 2021-07-13T00:20:32.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=754da02d-b699-4a8a-95ff-f4ad3bf7d788 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:32.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=754da02d-b699-4a8a-95ff-f4ad3bf7d788 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:35.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=82bacd3c-729f-473e-8ffb-f01178da6cb5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:35.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=82bacd3c-729f-473e-8ffb-f01178da6cb5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6b24ee82 2021-07-13T00:20:35.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=82bacd3c-729f-473e-8ffb-f01178da6cb5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:35.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=82bacd3c-729f-473e-8ffb-f01178da6cb5 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:37.917Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:20:37.928Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:20:38.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e766b6c9-4792-48a2-9ae0-72dc7c6d5a2c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:38.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e766b6c9-4792-48a2-9ae0-72dc7c6d5a2c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@29ee202 2021-07-13T00:20:38.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e766b6c9-4792-48a2-9ae0-72dc7c6d5a2c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:38.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e766b6c9-4792-48a2-9ae0-72dc7c6d5a2c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:41.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6015e931-8203-4976-a61b-adc85e653b93 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:41.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6015e931-8203-4976-a61b-adc85e653b93 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@59f2789 2021-07-13T00:20:41.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6015e931-8203-4976-a61b-adc85e653b93 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:41.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6015e931-8203-4976-a61b-adc85e653b93 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:45.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=422bb544-e9a0-408c-882b-9b2ff4376076 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:45.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=422bb544-e9a0-408c-882b-9b2ff4376076 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@94aea89 2021-07-13T00:20:45.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=422bb544-e9a0-408c-882b-9b2ff4376076 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:45.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=422bb544-e9a0-408c-882b-9b2ff4376076 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:46.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:20:46.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:20:48.070Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=422bb544-e9a0-408c-882b-9b2ff4376076 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:20:48.071Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=422bb544-e9a0-408c-882b-9b2ff4376076 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:20:48.071Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=422bb544-e9a0-408c-882b-9b2ff4376076 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:20:48.072Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=422bb544-e9a0-408c-882b-9b2ff4376076 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:20:49.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04e4b906-2efe-4ecb-b60b-629c3b7aba4e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:49.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04e4b906-2efe-4ecb-b60b-629c3b7aba4e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@455645dc 2021-07-13T00:20:49.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04e4b906-2efe-4ecb-b60b-629c3b7aba4e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:49.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04e4b906-2efe-4ecb-b60b-629c3b7aba4e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:49.913Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:20:50.662Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=04e4b906-2efe-4ecb-b60b-629c3b7aba4e ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:20:50.662Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=04e4b906-2efe-4ecb-b60b-629c3b7aba4e ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:20:51.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=932298fe-2184-4e4d-b7f5-36a92083aeb3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:51.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=932298fe-2184-4e4d-b7f5-36a92083aeb3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4c72eb65 2021-07-13T00:20:51.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=932298fe-2184-4e4d-b7f5-36a92083aeb3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:51.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=932298fe-2184-4e4d-b7f5-36a92083aeb3 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:52.565Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=932298fe-2184-4e4d-b7f5-36a92083aeb3 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:20:52.565Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=932298fe-2184-4e4d-b7f5-36a92083aeb3 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:20:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:20:52.955Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:20:53.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af69684a-bd95-44b2-9582-97fd7980d552 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:53.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af69684a-bd95-44b2-9582-97fd7980d552 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@12325ec0 2021-07-13T00:20:53.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af69684a-bd95-44b2-9582-97fd7980d552 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:53.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af69684a-bd95-44b2-9582-97fd7980d552 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:56.677Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73e7ec20-013a-4f98-a27a-94b9754a520c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:56.677Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73e7ec20-013a-4f98-a27a-94b9754a520c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@65bd81ff 2021-07-13T00:20:56.677Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73e7ec20-013a-4f98-a27a-94b9754a520c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:56.677Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73e7ec20-013a-4f98-a27a-94b9754a520c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:20:59.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61e0136d-4758-4ddb-80f0-ce931b0c7173 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:20:59.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61e0136d-4758-4ddb-80f0-ce931b0c7173 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3cc53dcf 2021-07-13T00:20:59.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61e0136d-4758-4ddb-80f0-ce931b0c7173 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:20:59.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61e0136d-4758-4ddb-80f0-ce931b0c7173 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:21:03.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3580dc2-5eac-4f5e-ae53-9432879695fa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:21:03.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3580dc2-5eac-4f5e-ae53-9432879695fa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6737d0f1 2021-07-13T00:21:03.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3580dc2-5eac-4f5e-ae53-9432879695fa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:21:03.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3580dc2-5eac-4f5e-ae53-9432879695fa PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:21:05.773Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc11c1fa-d91d-4323-8346-254b577f448c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:21:05.773Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc11c1fa-d91d-4323-8346-254b577f448c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@67735689 2021-07-13T00:21:05.773Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc11c1fa-d91d-4323-8346-254b577f448c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:21:05.773Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc11c1fa-d91d-4323-8346-254b577f448c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:21:07.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:21:07.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:21:09.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b4e57be-1820-4ae7-85bb-89472315ea21 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:21:09.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b4e57be-1820-4ae7-85bb-89472315ea21 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6ec891e1 2021-07-13T00:21:09.071Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b4e57be-1820-4ae7-85bb-89472315ea21 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:21:09.071Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b4e57be-1820-4ae7-85bb-89472315ea21 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:21:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:21:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:21:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:21:12.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f015ac3-24c1-41cc-805d-e3a6fefcb1d6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:21:12.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f015ac3-24c1-41cc-805d-e3a6fefcb1d6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7bf6f2b2 2021-07-13T00:21:12.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f015ac3-24c1-41cc-805d-e3a6fefcb1d6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:21:12.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f015ac3-24c1-41cc-805d-e3a6fefcb1d6 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:21:15.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3338cfe-8c0f-4f9c-ac63-182a7dcfd698 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:21:15.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3338cfe-8c0f-4f9c-ac63-182a7dcfd698 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@221f1565 2021-07-13T00:21:15.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3338cfe-8c0f-4f9c-ac63-182a7dcfd698 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:21:15.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3338cfe-8c0f-4f9c-ac63-182a7dcfd698 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:21:17.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=39acae85-b1fa-422d-9864-e96b0b15f73b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:21:17.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=39acae85-b1fa-422d-9864-e96b0b15f73b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f25c8b1 2021-07-13T00:21:17.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=39acae85-b1fa-422d-9864-e96b0b15f73b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:21:17.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=39acae85-b1fa-422d-9864-e96b0b15f73b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:21:20.872Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=39acae85-b1fa-422d-9864-e96b0b15f73b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:21:20.873Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=39acae85-b1fa-422d-9864-e96b0b15f73b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:21:20.874Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=39acae85-b1fa-422d-9864-e96b0b15f73b ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:21:20.874Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=39acae85-b1fa-422d-9864-e96b0b15f73b ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:21:21.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a43e78fb-1e78-42a0-ae8d-32e73776e3b9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:21:21.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a43e78fb-1e78-42a0-ae8d-32e73776e3b9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@75480d2d 2021-07-13T00:21:21.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a43e78fb-1e78-42a0-ae8d-32e73776e3b9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:21:21.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a43e78fb-1e78-42a0-ae8d-32e73776e3b9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:21:22.462Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=a43e78fb-1e78-42a0-ae8d-32e73776e3b9 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:21:22.462Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=a43e78fb-1e78-42a0-ae8d-32e73776e3b9 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:21:22.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:21:22.929Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:21:23.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f011bbd-236a-4118-948c-b78b3d67e314 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:21:23.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f011bbd-236a-4118-948c-b78b3d67e314 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63899c30 2021-07-13T00:21:23.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f011bbd-236a-4118-948c-b78b3d67e314 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:21:23.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f011bbd-236a-4118-948c-b78b3d67e314 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:21:24.876Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:21:26.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca5ba84d-41da-4423-8f72-5d5fb72a39aa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:21:26.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca5ba84d-41da-4423-8f72-5d5fb72a39aa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1218a0b0 2021-07-13T00:21:26.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca5ba84d-41da-4423-8f72-5d5fb72a39aa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:21:26.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca5ba84d-41da-4423-8f72-5d5fb72a39aa PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:21:29.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9959827f-4f81-408a-a3eb-f0e97720bda8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:21:29.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9959827f-4f81-408a-a3eb-f0e97720bda8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@278ce6bb 2021-07-13T00:21:29.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9959827f-4f81-408a-a3eb-f0e97720bda8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:21:29.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9959827f-4f81-408a-a3eb-f0e97720bda8 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:21:32.667Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=9959827f-4f81-408a-a3eb-f0e97720bda8 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:21:32.667Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=9959827f-4f81-408a-a3eb-f0e97720bda8 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:21:32.668Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9959827f-4f81-408a-a3eb-f0e97720bda8 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:21:32.668Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9959827f-4f81-408a-a3eb-f0e97720bda8 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:21:33.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=574c6f59-4b3b-43dd-ace2-7f762ae0cd42 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:21:33.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=574c6f59-4b3b-43dd-ace2-7f762ae0cd42 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@26f835ac 2021-07-13T00:21:33.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=574c6f59-4b3b-43dd-ace2-7f762ae0cd42 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:21:33.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=574c6f59-4b3b-43dd-ace2-7f762ae0cd42 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:21:34.661Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=574c6f59-4b3b-43dd-ace2-7f762ae0cd42 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:21:34.661Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=574c6f59-4b3b-43dd-ace2-7f762ae0cd42 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:21:35.670Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9817374d-98be-4efa-98d4-6b438528d710 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:21:35.670Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9817374d-98be-4efa-98d4-6b438528d710 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5ef569e8 2021-07-13T00:21:35.670Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9817374d-98be-4efa-98d4-6b438528d710 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:21:35.670Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9817374d-98be-4efa-98d4-6b438528d710 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:21:37.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:21:37.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:21:38.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b65693ae-0bf7-4efc-83f4-491daf91ef18 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:21:38.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b65693ae-0bf7-4efc-83f4-491daf91ef18 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1e8cf5b3 2021-07-13T00:21:38.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b65693ae-0bf7-4efc-83f4-491daf91ef18 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:21:38.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b65693ae-0bf7-4efc-83f4-491daf91ef18 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:21:41.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=12c79c78-be12-42ce-8fe2-042801946b7b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:21:41.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=12c79c78-be12-42ce-8fe2-042801946b7b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@79bef69b 2021-07-13T00:21:41.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=12c79c78-be12-42ce-8fe2-042801946b7b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:21:41.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=12c79c78-be12-42ce-8fe2-042801946b7b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:21:44.864Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=12c79c78-be12-42ce-8fe2-042801946b7b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:21:44.864Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=12c79c78-be12-42ce-8fe2-042801946b7b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:21:44.864Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=12c79c78-be12-42ce-8fe2-042801946b7b ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:21:44.865Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=12c79c78-be12-42ce-8fe2-042801946b7b ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:21:45.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b52f183b-90ab-4387-a3b1-5e74b62f77ba PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:21:45.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b52f183b-90ab-4387-a3b1-5e74b62f77ba PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@ad8c15e 2021-07-13T00:21:45.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b52f183b-90ab-4387-a3b1-5e74b62f77ba PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:21:45.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b52f183b-90ab-4387-a3b1-5e74b62f77ba PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:21:46.676Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b52f183b-90ab-4387-a3b1-5e74b62f77ba ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:21:46.676Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b52f183b-90ab-4387-a3b1-5e74b62f77ba ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:21:48.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c26ff34-5e72-46ef-8e10-4c28ab578dd3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:21:48.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c26ff34-5e72-46ef-8e10-4c28ab578dd3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64f10f8 2021-07-13T00:21:48.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c26ff34-5e72-46ef-8e10-4c28ab578dd3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:21:48.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c26ff34-5e72-46ef-8e10-4c28ab578dd3 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:21:50.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc556231-8089-4c7b-aa8d-b209c7ec1385 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:21:50.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc556231-8089-4c7b-aa8d-b209c7ec1385 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@8cb2484 2021-07-13T00:21:50.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc556231-8089-4c7b-aa8d-b209c7ec1385 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:21:50.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc556231-8089-4c7b-aa8d-b209c7ec1385 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:21:52.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:21:52.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:21:53.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3acf4c1-7c42-4f2f-b9cc-95967a74befa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:21:53.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3acf4c1-7c42-4f2f-b9cc-95967a74befa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4455c6f2 2021-07-13T00:21:53.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3acf4c1-7c42-4f2f-b9cc-95967a74befa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:21:53.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3acf4c1-7c42-4f2f-b9cc-95967a74befa PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:21:56.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=884864e0-2c98-4b7e-8f43-d11830985066 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:21:56.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=884864e0-2c98-4b7e-8f43-d11830985066 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@119a4df 2021-07-13T00:21:56.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=884864e0-2c98-4b7e-8f43-d11830985066 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:21:56.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=884864e0-2c98-4b7e-8f43-d11830985066 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:22:00.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6c8a11a-6781-4be8-9f28-c6f36bbb0cbb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:22:00.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6c8a11a-6781-4be8-9f28-c6f36bbb0cbb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f2fbcd4 2021-07-13T00:22:00.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6c8a11a-6781-4be8-9f28-c6f36bbb0cbb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:22:00.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6c8a11a-6781-4be8-9f28-c6f36bbb0cbb PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:22:02.774Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7095189-a3d0-4e32-a80a-dc33b3ce60b0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:22:02.774Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7095189-a3d0-4e32-a80a-dc33b3ce60b0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2e9148b1 2021-07-13T00:22:02.774Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7095189-a3d0-4e32-a80a-dc33b3ce60b0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:22:02.774Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7095189-a3d0-4e32-a80a-dc33b3ce60b0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:22:05.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dbc1d7bb-de25-4903-94be-2cbc3026854e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:22:05.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dbc1d7bb-de25-4903-94be-2cbc3026854e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@441969c7 2021-07-13T00:22:05.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dbc1d7bb-de25-4903-94be-2cbc3026854e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:22:05.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dbc1d7bb-de25-4903-94be-2cbc3026854e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:22:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:22:07.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:22:08.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a47594e-664e-4ebe-a129-3838b68da90e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:22:08.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a47594e-664e-4ebe-a129-3838b68da90e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@8ea02c7 2021-07-13T00:22:08.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a47594e-664e-4ebe-a129-3838b68da90e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:22:08.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a47594e-664e-4ebe-a129-3838b68da90e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:22:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:22:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:22:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:22:11.675Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69cb331b-d112-41ef-8700-a4f1406581af PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:22:11.675Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69cb331b-d112-41ef-8700-a4f1406581af PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@aa1c612 2021-07-13T00:22:11.675Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69cb331b-d112-41ef-8700-a4f1406581af PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:22:11.675Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=69cb331b-d112-41ef-8700-a4f1406581af PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:22:15.157Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7c02be0-d38d-4874-9c65-c4a5a5deb7fe PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:22:15.157Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7c02be0-d38d-4874-9c65-c4a5a5deb7fe PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2e51d337 2021-07-13T00:22:15.157Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7c02be0-d38d-4874-9c65-c4a5a5deb7fe PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:22:15.157Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7c02be0-d38d-4874-9c65-c4a5a5deb7fe PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:22:18.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=905895ba-e343-408f-a82c-a95957d664bf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:22:18.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=905895ba-e343-408f-a82c-a95957d664bf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2189d06b 2021-07-13T00:22:18.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=905895ba-e343-408f-a82c-a95957d664bf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:22:18.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=905895ba-e343-408f-a82c-a95957d664bf PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:22:21.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f9c2770-c072-4894-869d-5e7a151df418 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:22:21.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f9c2770-c072-4894-869d-5e7a151df418 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@79ad02a7 2021-07-13T00:22:21.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f9c2770-c072-4894-869d-5e7a151df418 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:22:21.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f9c2770-c072-4894-869d-5e7a151df418 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:22:22.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:22:22.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:22:23.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e4abd2a-ea6c-4844-a05e-e4247399ad4d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:22:23.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e4abd2a-ea6c-4844-a05e-e4247399ad4d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7d38cad6 2021-07-13T00:22:23.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e4abd2a-ea6c-4844-a05e-e4247399ad4d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:22:23.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e4abd2a-ea6c-4844-a05e-e4247399ad4d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:22:25.369Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:22:26.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1aa72fc2-0880-4dfb-bc0e-3ddb1d2ab123 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:22:26.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1aa72fc2-0880-4dfb-bc0e-3ddb1d2ab123 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5d50662d 2021-07-13T00:22:26.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1aa72fc2-0880-4dfb-bc0e-3ddb1d2ab123 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:22:26.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1aa72fc2-0880-4dfb-bc0e-3ddb1d2ab123 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:22:30.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=77d2b96d-6f79-43b6-a41a-2caee3eb9b4a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:22:30.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=77d2b96d-6f79-43b6-a41a-2caee3eb9b4a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@32b5ff64 2021-07-13T00:22:30.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=77d2b96d-6f79-43b6-a41a-2caee3eb9b4a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:22:30.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=77d2b96d-6f79-43b6-a41a-2caee3eb9b4a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:22:32.681Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a6dab02-ef95-4f63-b66b-1e4842d7ff47 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:22:32.681Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a6dab02-ef95-4f63-b66b-1e4842d7ff47 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@231c7baf 2021-07-13T00:22:32.681Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a6dab02-ef95-4f63-b66b-1e4842d7ff47 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:22:32.681Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a6dab02-ef95-4f63-b66b-1e4842d7ff47 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:22:35.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91356248-3cea-421e-9ac5-86d90e514a3d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:22:35.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91356248-3cea-421e-9ac5-86d90e514a3d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@18c96b60 2021-07-13T00:22:35.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91356248-3cea-421e-9ac5-86d90e514a3d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:22:35.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91356248-3cea-421e-9ac5-86d90e514a3d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:22:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:22:37.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:22:38.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c30f5ea6-8598-4757-879b-7c33c71160e5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:22:38.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c30f5ea6-8598-4757-879b-7c33c71160e5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@46f6175a 2021-07-13T00:22:38.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c30f5ea6-8598-4757-879b-7c33c71160e5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:22:38.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c30f5ea6-8598-4757-879b-7c33c71160e5 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:22:40.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:22:40.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:22:41.769Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=c30f5ea6-8598-4757-879b-7c33c71160e5 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:22:41.769Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=c30f5ea6-8598-4757-879b-7c33c71160e5 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:22:41.770Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c30f5ea6-8598-4757-879b-7c33c71160e5 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:22:41.770Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c30f5ea6-8598-4757-879b-7c33c71160e5 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:22:42.563Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:22:42.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d17497f-13f9-4ddb-ada8-6cf0adc48c77 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:22:42.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d17497f-13f9-4ddb-ada8-6cf0adc48c77 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5b259e3f 2021-07-13T00:22:42.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d17497f-13f9-4ddb-ada8-6cf0adc48c77 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:22:42.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d17497f-13f9-4ddb-ada8-6cf0adc48c77 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:22:43.831Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=3d17497f-13f9-4ddb-ada8-6cf0adc48c77 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:22:43.832Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=3d17497f-13f9-4ddb-ada8-6cf0adc48c77 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:22:45.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1a15bb6-30a5-4276-bc8b-63287b47d7cb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:22:45.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1a15bb6-30a5-4276-bc8b-63287b47d7cb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4865bd98 2021-07-13T00:22:45.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1a15bb6-30a5-4276-bc8b-63287b47d7cb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:22:45.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1a15bb6-30a5-4276-bc8b-63287b47d7cb PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:22:46.266Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=e1a15bb6-30a5-4276-bc8b-63287b47d7cb ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:22:46.266Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=e1a15bb6-30a5-4276-bc8b-63287b47d7cb ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:22:47.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=746e036b-c4ac-4113-a263-392c0c4b6a25 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:22:47.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=746e036b-c4ac-4113-a263-392c0c4b6a25 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3e5f3f09 2021-07-13T00:22:47.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=746e036b-c4ac-4113-a263-392c0c4b6a25 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:22:47.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=746e036b-c4ac-4113-a263-392c0c4b6a25 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:22:51.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b91089ba-80bf-4c22-b1ca-b222cab47417 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:22:51.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b91089ba-80bf-4c22-b1ca-b222cab47417 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@49fc8a9b 2021-07-13T00:22:51.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b91089ba-80bf-4c22-b1ca-b222cab47417 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:22:51.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b91089ba-80bf-4c22-b1ca-b222cab47417 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:22:52.918Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:22:52.945Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:22:53.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=109e277a-d61d-47a9-bdc4-8e9c7a748bd4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:22:53.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=109e277a-d61d-47a9-bdc4-8e9c7a748bd4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@70def7bc 2021-07-13T00:22:53.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=109e277a-d61d-47a9-bdc4-8e9c7a748bd4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:22:53.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=109e277a-d61d-47a9-bdc4-8e9c7a748bd4 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:22:56.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d90070e2-3b16-43d9-b3fb-636e13e2e391 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:22:56.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d90070e2-3b16-43d9-b3fb-636e13e2e391 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@15321d42 2021-07-13T00:22:56.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d90070e2-3b16-43d9-b3fb-636e13e2e391 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:22:56.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d90070e2-3b16-43d9-b3fb-636e13e2e391 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:22:59.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=749ecf26-8a38-4220-bf06-165c6bbd3cbb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:22:59.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=749ecf26-8a38-4220-bf06-165c6bbd3cbb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5252df9a 2021-07-13T00:22:59.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=749ecf26-8a38-4220-bf06-165c6bbd3cbb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:22:59.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=749ecf26-8a38-4220-bf06-165c6bbd3cbb PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:23:02.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7581bbbd-7f9f-418e-92b4-0f593c5e9170 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:23:02.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7581bbbd-7f9f-418e-92b4-0f593c5e9170 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f572f19 2021-07-13T00:23:02.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7581bbbd-7f9f-418e-92b4-0f593c5e9170 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:23:02.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7581bbbd-7f9f-418e-92b4-0f593c5e9170 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:23:05.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ced1d394-a001-41ce-8d07-d300ddad3469 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:23:05.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ced1d394-a001-41ce-8d07-d300ddad3469 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78334458 2021-07-13T00:23:05.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ced1d394-a001-41ce-8d07-d300ddad3469 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:23:05.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ced1d394-a001-41ce-8d07-d300ddad3469 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:23:07.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:23:07.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:23:09.267Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f654fb9-2b34-4a69-a769-491392e819dc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:23:09.267Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f654fb9-2b34-4a69-a769-491392e819dc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7dee8e11 2021-07-13T00:23:09.267Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f654fb9-2b34-4a69-a769-491392e819dc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:23:09.267Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f654fb9-2b34-4a69-a769-491392e819dc PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:23:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:23:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:23:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:23:12.274Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=0f654fb9-2b34-4a69-a769-491392e819dc ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:23:12.275Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=0f654fb9-2b34-4a69-a769-491392e819dc ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:23:12.275Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=0f654fb9-2b34-4a69-a769-491392e819dc ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:23:12.275Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=0f654fb9-2b34-4a69-a769-491392e819dc ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:23:13.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=630fb245-f613-480f-ac80-095821b61c29 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:23:13.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=630fb245-f613-480f-ac80-095821b61c29 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@39843039 2021-07-13T00:23:13.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=630fb245-f613-480f-ac80-095821b61c29 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:23:13.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=630fb245-f613-480f-ac80-095821b61c29 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:23:15.666Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=630fb245-f613-480f-ac80-095821b61c29 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:23:15.666Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=630fb245-f613-480f-ac80-095821b61c29 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:23:16.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b7d64bf0-1b55-4bc6-8469-9a32c3dd1abc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:23:16.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b7d64bf0-1b55-4bc6-8469-9a32c3dd1abc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1d9e65d6 2021-07-13T00:23:16.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b7d64bf0-1b55-4bc6-8469-9a32c3dd1abc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:23:16.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b7d64bf0-1b55-4bc6-8469-9a32c3dd1abc PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:23:18.770Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7571235-df38-486f-8db4-4dffc2733d8c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:23:18.770Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7571235-df38-486f-8db4-4dffc2733d8c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1489fe72 2021-07-13T00:23:18.770Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7571235-df38-486f-8db4-4dffc2733d8c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:23:18.770Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7571235-df38-486f-8db4-4dffc2733d8c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:23:21.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=667d0e18-1d0c-4bea-9119-42f14975276b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:23:21.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=667d0e18-1d0c-4bea-9119-42f14975276b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@c941122 2021-07-13T00:23:21.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=667d0e18-1d0c-4bea-9119-42f14975276b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:23:21.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=667d0e18-1d0c-4bea-9119-42f14975276b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:23:22.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:23:22.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:23:23.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34ddecc8-a479-47b2-984c-52b7c8e4c029 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:23:23.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34ddecc8-a479-47b2-984c-52b7c8e4c029 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c965500 2021-07-13T00:23:23.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34ddecc8-a479-47b2-984c-52b7c8e4c029 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:23:23.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34ddecc8-a479-47b2-984c-52b7c8e4c029 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:23:24.704Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:23:26.766Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=34ddecc8-a479-47b2-984c-52b7c8e4c029 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:23:26.766Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=34ddecc8-a479-47b2-984c-52b7c8e4c029 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:23:26.766Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=34ddecc8-a479-47b2-984c-52b7c8e4c029 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:23:26.767Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=34ddecc8-a479-47b2-984c-52b7c8e4c029 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:23:27.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7feffa9a-8668-4aa8-afd9-822703799345 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:23:27.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7feffa9a-8668-4aa8-afd9-822703799345 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@72c82391 2021-07-13T00:23:27.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7feffa9a-8668-4aa8-afd9-822703799345 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:23:27.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7feffa9a-8668-4aa8-afd9-822703799345 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:23:28.361Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=7feffa9a-8668-4aa8-afd9-822703799345 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:23:28.361Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=7feffa9a-8668-4aa8-afd9-822703799345 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:23:29.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33a2b6d6-80dd-4eb0-adaa-38efa5687bbb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:23:29.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33a2b6d6-80dd-4eb0-adaa-38efa5687bbb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@55830da 2021-07-13T00:23:29.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33a2b6d6-80dd-4eb0-adaa-38efa5687bbb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:23:29.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33a2b6d6-80dd-4eb0-adaa-38efa5687bbb PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:23:32.871Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c42a056d-c97a-40b8-88f8-41f2f7908439 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:23:32.871Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c42a056d-c97a-40b8-88f8-41f2f7908439 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2b484b24 2021-07-13T00:23:32.871Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c42a056d-c97a-40b8-88f8-41f2f7908439 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:23:32.871Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c42a056d-c97a-40b8-88f8-41f2f7908439 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:23:35.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f4d6ca2-5d88-4117-9aee-21aaf5b959ec PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:23:35.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f4d6ca2-5d88-4117-9aee-21aaf5b959ec PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@18af2177 2021-07-13T00:23:35.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f4d6ca2-5d88-4117-9aee-21aaf5b959ec PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:23:35.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f4d6ca2-5d88-4117-9aee-21aaf5b959ec PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:23:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:23:37.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:23:39.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d309b65e-a5db-4ebb-b17e-da97d362ee2d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:23:39.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d309b65e-a5db-4ebb-b17e-da97d362ee2d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78e80c2f 2021-07-13T00:23:39.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d309b65e-a5db-4ebb-b17e-da97d362ee2d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:23:39.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d309b65e-a5db-4ebb-b17e-da97d362ee2d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:23:41.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=291d7aef-dee4-4a84-bd0e-4ccda61653e0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:23:41.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=291d7aef-dee4-4a84-bd0e-4ccda61653e0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3c723d08 2021-07-13T00:23:41.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=291d7aef-dee4-4a84-bd0e-4ccda61653e0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:23:41.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=291d7aef-dee4-4a84-bd0e-4ccda61653e0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:23:43.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:23:43.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:23:44.769Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=291d7aef-dee4-4a84-bd0e-4ccda61653e0 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:23:44.769Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=291d7aef-dee4-4a84-bd0e-4ccda61653e0 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:23:44.769Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=291d7aef-dee4-4a84-bd0e-4ccda61653e0 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:23:44.769Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=291d7aef-dee4-4a84-bd0e-4ccda61653e0 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:23:45.764Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:23:46.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33dd13a3-fc7b-4516-89df-d947266dac10 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:23:46.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33dd13a3-fc7b-4516-89df-d947266dac10 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@558a40c9 2021-07-13T00:23:46.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33dd13a3-fc7b-4516-89df-d947266dac10 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:23:46.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33dd13a3-fc7b-4516-89df-d947266dac10 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:23:47.018Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=33dd13a3-fc7b-4516-89df-d947266dac10 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:23:47.018Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=33dd13a3-fc7b-4516-89df-d947266dac10 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:23:48.367Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2c8f818-4925-41f1-88d5-66af5eedee93 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:23:48.367Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2c8f818-4925-41f1-88d5-66af5eedee93 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@c7e9c54 2021-07-13T00:23:48.367Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2c8f818-4925-41f1-88d5-66af5eedee93 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:23:48.367Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2c8f818-4925-41f1-88d5-66af5eedee93 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:23:48.963Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b2c8f818-4925-41f1-88d5-66af5eedee93 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:23:48.963Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b2c8f818-4925-41f1-88d5-66af5eedee93 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:23:50.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ebd281e1-4806-42bd-a460-53bea43e1001 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:23:50.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ebd281e1-4806-42bd-a460-53bea43e1001 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@21497aae 2021-07-13T00:23:50.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ebd281e1-4806-42bd-a460-53bea43e1001 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:23:50.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ebd281e1-4806-42bd-a460-53bea43e1001 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:23:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:23:52.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:23:53.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a836cc2-339a-436c-be3c-af7a3bfc2caa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:23:53.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a836cc2-339a-436c-be3c-af7a3bfc2caa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2642ea46 2021-07-13T00:23:53.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a836cc2-339a-436c-be3c-af7a3bfc2caa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:23:53.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a836cc2-339a-436c-be3c-af7a3bfc2caa PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:23:56.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a832b189-54c6-40cd-af62-b5951f5af614 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:23:56.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a832b189-54c6-40cd-af62-b5951f5af614 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@69fecacf 2021-07-13T00:23:56.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a832b189-54c6-40cd-af62-b5951f5af614 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:23:56.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a832b189-54c6-40cd-af62-b5951f5af614 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:24:00.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db201936-5329-4fcd-8a08-683a21d635cc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:24:00.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db201936-5329-4fcd-8a08-683a21d635cc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@119a6798 2021-07-13T00:24:00.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db201936-5329-4fcd-8a08-683a21d635cc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:24:00.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db201936-5329-4fcd-8a08-683a21d635cc PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:24:02.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ea01c54-4950-41b2-a0ac-68492cfef750 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:24:02.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ea01c54-4950-41b2-a0ac-68492cfef750 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7bece30f 2021-07-13T00:24:02.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ea01c54-4950-41b2-a0ac-68492cfef750 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:24:02.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ea01c54-4950-41b2-a0ac-68492cfef750 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:24:05.672Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e52c55c-c372-4be1-bdf5-40e5f064d4d8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:24:05.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e52c55c-c372-4be1-bdf5-40e5f064d4d8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4acd26ee 2021-07-13T00:24:05.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e52c55c-c372-4be1-bdf5-40e5f064d4d8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:24:05.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e52c55c-c372-4be1-bdf5-40e5f064d4d8 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:24:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:24:07.988Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:24:08.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08da3904-e60d-40c9-a8cb-4f6fa9fe6b4e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:24:08.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08da3904-e60d-40c9-a8cb-4f6fa9fe6b4e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@55fa35ca 2021-07-13T00:24:08.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08da3904-e60d-40c9-a8cb-4f6fa9fe6b4e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:24:08.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08da3904-e60d-40c9-a8cb-4f6fa9fe6b4e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:24:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:24:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:24:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:24:12.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=439e8a4c-de17-4e7b-8d02-1443d617a3ff PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:24:12.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=439e8a4c-de17-4e7b-8d02-1443d617a3ff PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@171e2e4a 2021-07-13T00:24:12.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=439e8a4c-de17-4e7b-8d02-1443d617a3ff PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:24:12.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=439e8a4c-de17-4e7b-8d02-1443d617a3ff PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:24:14.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0c533be-1bc4-4066-adaf-fa197dc25fcd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:24:14.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0c533be-1bc4-4066-adaf-fa197dc25fcd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@bf6533d 2021-07-13T00:24:14.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0c533be-1bc4-4066-adaf-fa197dc25fcd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:24:14.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c0c533be-1bc4-4066-adaf-fa197dc25fcd PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:24:17.556Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2137d49-f5ba-4bfb-8d13-3103619e5e5c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:24:17.556Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2137d49-f5ba-4bfb-8d13-3103619e5e5c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@282feefb 2021-07-13T00:24:17.556Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2137d49-f5ba-4bfb-8d13-3103619e5e5c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:24:17.556Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2137d49-f5ba-4bfb-8d13-3103619e5e5c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:24:20.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=65dcc4d5-693e-46a4-b213-a802735d9c58 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:24:20.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=65dcc4d5-693e-46a4-b213-a802735d9c58 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2c7fca51 2021-07-13T00:24:20.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=65dcc4d5-693e-46a4-b213-a802735d9c58 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:24:20.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=65dcc4d5-693e-46a4-b213-a802735d9c58 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:24:22.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:24:22.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:24:23.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=981d51d6-2c3c-41f4-a6f2-7bbdc7a3ed21 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:24:23.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=981d51d6-2c3c-41f4-a6f2-7bbdc7a3ed21 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4a75cdea 2021-07-13T00:24:23.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=981d51d6-2c3c-41f4-a6f2-7bbdc7a3ed21 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:24:23.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=981d51d6-2c3c-41f4-a6f2-7bbdc7a3ed21 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:24:24.705Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:24:26.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2fdbdd15-e1c0-488a-9f1b-f984c4b1e8f5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:24:26.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2fdbdd15-e1c0-488a-9f1b-f984c4b1e8f5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@73230022 2021-07-13T00:24:26.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2fdbdd15-e1c0-488a-9f1b-f984c4b1e8f5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:24:26.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2fdbdd15-e1c0-488a-9f1b-f984c4b1e8f5 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:24:29.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f40aa11-79be-450b-a2bf-406bfa584848 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:24:29.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f40aa11-79be-450b-a2bf-406bfa584848 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@24ac708b 2021-07-13T00:24:29.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f40aa11-79be-450b-a2bf-406bfa584848 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:24:29.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f40aa11-79be-450b-a2bf-406bfa584848 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:24:32.857Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62bc99d4-15ba-4947-be9e-128fe08f347e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:24:32.857Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62bc99d4-15ba-4947-be9e-128fe08f347e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@76d31b0d 2021-07-13T00:24:32.857Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62bc99d4-15ba-4947-be9e-128fe08f347e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:24:32.857Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62bc99d4-15ba-4947-be9e-128fe08f347e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:24:35.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2b9283c-bd13-4d26-81f5-2a0e18d3547b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:24:35.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2b9283c-bd13-4d26-81f5-2a0e18d3547b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@12ad92a0 2021-07-13T00:24:35.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2b9283c-bd13-4d26-81f5-2a0e18d3547b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:24:35.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2b9283c-bd13-4d26-81f5-2a0e18d3547b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:24:37.918Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:24:37.932Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:24:38.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fe16f78-7702-4fbb-8210-c316f8e04ab3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:24:38.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fe16f78-7702-4fbb-8210-c316f8e04ab3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7964a2c7 2021-07-13T00:24:38.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fe16f78-7702-4fbb-8210-c316f8e04ab3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:24:38.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fe16f78-7702-4fbb-8210-c316f8e04ab3 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:24:42.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52bc8744-465a-4e86-8ecb-053e9b4fedf7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:24:42.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52bc8744-465a-4e86-8ecb-053e9b4fedf7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3e0bbd48 2021-07-13T00:24:42.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52bc8744-465a-4e86-8ecb-053e9b4fedf7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:24:42.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52bc8744-465a-4e86-8ecb-053e9b4fedf7 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:24:44.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52ebe275-9c23-4636-b87b-4caf9f2e91cb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:24:44.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52ebe275-9c23-4636-b87b-4caf9f2e91cb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@50eed2f7 2021-07-13T00:24:44.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52ebe275-9c23-4636-b87b-4caf9f2e91cb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:24:44.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52ebe275-9c23-4636-b87b-4caf9f2e91cb PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:24:48.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3667654c-3bc8-41e1-9a41-0855ed78de76 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:24:48.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3667654c-3bc8-41e1-9a41-0855ed78de76 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ea872d8 2021-07-13T00:24:48.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3667654c-3bc8-41e1-9a41-0855ed78de76 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:24:48.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3667654c-3bc8-41e1-9a41-0855ed78de76 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:24:51.564Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15ef0c65-e7c4-4e8e-9242-65a633e70ccd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:24:51.564Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15ef0c65-e7c4-4e8e-9242-65a633e70ccd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@72069f50 2021-07-13T00:24:51.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15ef0c65-e7c4-4e8e-9242-65a633e70ccd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:24:51.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15ef0c65-e7c4-4e8e-9242-65a633e70ccd PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:24:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:24:52.928Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:24:53.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36c180d2-36a7-49e4-af85-95e74d4c4ec4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:24:53.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36c180d2-36a7-49e4-af85-95e74d4c4ec4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@11049da5 2021-07-13T00:24:53.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36c180d2-36a7-49e4-af85-95e74d4c4ec4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:24:53.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36c180d2-36a7-49e4-af85-95e74d4c4ec4 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:24:56.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e61c740a-7005-4797-8c0d-42e280e1aae0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:24:56.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e61c740a-7005-4797-8c0d-42e280e1aae0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@375b8c18 2021-07-13T00:24:56.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e61c740a-7005-4797-8c0d-42e280e1aae0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:24:56.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e61c740a-7005-4797-8c0d-42e280e1aae0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:24:59.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33d8f6c3-9aac-48b7-bc8a-69a530f87c38 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:24:59.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33d8f6c3-9aac-48b7-bc8a-69a530f87c38 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5a080699 2021-07-13T00:24:59.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33d8f6c3-9aac-48b7-bc8a-69a530f87c38 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:24:59.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33d8f6c3-9aac-48b7-bc8a-69a530f87c38 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:25:03.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2771e75-eb32-4db9-addb-543771101a7e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:25:03.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2771e75-eb32-4db9-addb-543771101a7e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c32ee97 2021-07-13T00:25:03.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2771e75-eb32-4db9-addb-543771101a7e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:25:03.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2771e75-eb32-4db9-addb-543771101a7e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:25:05.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06dbf26f-2cdd-4490-b622-88c6f77db439 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:25:05.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06dbf26f-2cdd-4490-b622-88c6f77db439 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@29eea101 2021-07-13T00:25:05.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06dbf26f-2cdd-4490-b622-88c6f77db439 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:25:05.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=06dbf26f-2cdd-4490-b622-88c6f77db439 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:25:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:25:07.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:25:08.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1fccc7af-9ede-4910-a7c6-7195c40f5a87 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:25:08.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1fccc7af-9ede-4910-a7c6-7195c40f5a87 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@423ec368 2021-07-13T00:25:08.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1fccc7af-9ede-4910-a7c6-7195c40f5a87 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:25:08.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1fccc7af-9ede-4910-a7c6-7195c40f5a87 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:25:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:25:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:25:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:25:11.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f5d2ae9-7bc1-4db8-abf8-63e095948f94 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:25:11.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f5d2ae9-7bc1-4db8-abf8-63e095948f94 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@58183978 2021-07-13T00:25:11.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f5d2ae9-7bc1-4db8-abf8-63e095948f94 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:25:11.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f5d2ae9-7bc1-4db8-abf8-63e095948f94 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:25:15.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=14e52def-a68d-4bf4-8f2e-53ce71ee0adf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:25:15.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=14e52def-a68d-4bf4-8f2e-53ce71ee0adf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@535522ba 2021-07-13T00:25:15.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=14e52def-a68d-4bf4-8f2e-53ce71ee0adf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:25:15.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=14e52def-a68d-4bf4-8f2e-53ce71ee0adf PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:25:17.672Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41f47886-6da2-4fd5-9113-723e281e1ba5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:25:17.672Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41f47886-6da2-4fd5-9113-723e281e1ba5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@f6f6778 2021-07-13T00:25:17.672Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41f47886-6da2-4fd5-9113-723e281e1ba5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:25:17.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41f47886-6da2-4fd5-9113-723e281e1ba5 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:25:21.360Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0545d6a1-6107-4971-8499-c8769dee11ec PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:25:21.360Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0545d6a1-6107-4971-8499-c8769dee11ec PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3bfed07e 2021-07-13T00:25:21.360Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0545d6a1-6107-4971-8499-c8769dee11ec PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:25:21.360Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0545d6a1-6107-4971-8499-c8769dee11ec PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:25:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:25:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:25:23.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bfbe5df6-de45-4dd7-9c4c-473ecdd5ff8c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:25:23.461Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bfbe5df6-de45-4dd7-9c4c-473ecdd5ff8c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@22b4fd80 2021-07-13T00:25:23.461Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bfbe5df6-de45-4dd7-9c4c-473ecdd5ff8c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:25:23.461Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bfbe5df6-de45-4dd7-9c4c-473ecdd5ff8c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:25:24.779Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:25:27.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21f9c33b-ee41-484b-ab06-863b9f6b8d02 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:25:27.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21f9c33b-ee41-484b-ab06-863b9f6b8d02 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@580e5b9e 2021-07-13T00:25:27.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21f9c33b-ee41-484b-ab06-863b9f6b8d02 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:25:27.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21f9c33b-ee41-484b-ab06-863b9f6b8d02 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:25:30.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94809601-0d66-4061-8cfc-3ef49c3613cd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:25:30.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94809601-0d66-4061-8cfc-3ef49c3613cd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2caea340 2021-07-13T00:25:30.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94809601-0d66-4061-8cfc-3ef49c3613cd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:25:30.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94809601-0d66-4061-8cfc-3ef49c3613cd PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:25:32.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5075c075-dc7d-4513-b3ee-1b43dd1cf4d6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:25:32.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5075c075-dc7d-4513-b3ee-1b43dd1cf4d6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@611eeb8b 2021-07-13T00:25:32.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5075c075-dc7d-4513-b3ee-1b43dd1cf4d6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:25:32.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5075c075-dc7d-4513-b3ee-1b43dd1cf4d6 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:25:35.867Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=5075c075-dc7d-4513-b3ee-1b43dd1cf4d6 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:25:35.869Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=5075c075-dc7d-4513-b3ee-1b43dd1cf4d6 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:25:35.869Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=5075c075-dc7d-4513-b3ee-1b43dd1cf4d6 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:25:35.869Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=5075c075-dc7d-4513-b3ee-1b43dd1cf4d6 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:25:37.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bda0c68a-8c8b-4b68-9602-bb17c84ea4d2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:25:37.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bda0c68a-8c8b-4b68-9602-bb17c84ea4d2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@644342d5 2021-07-13T00:25:37.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bda0c68a-8c8b-4b68-9602-bb17c84ea4d2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:25:37.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bda0c68a-8c8b-4b68-9602-bb17c84ea4d2 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:25:37.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:25:37.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:25:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:25:37.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:25:40.121Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=bda0c68a-8c8b-4b68-9602-bb17c84ea4d2 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:25:40.122Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=bda0c68a-8c8b-4b68-9602-bb17c84ea4d2 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:25:40.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:25:41.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09a09c18-4903-4b49-817c-f8f634c5060e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:25:41.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09a09c18-4903-4b49-817c-f8f634c5060e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@17d6295c 2021-07-13T00:25:41.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09a09c18-4903-4b49-817c-f8f634c5060e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:25:41.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09a09c18-4903-4b49-817c-f8f634c5060e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:25:42.076Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=09a09c18-4903-4b49-817c-f8f634c5060e ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:25:42.076Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=09a09c18-4903-4b49-817c-f8f634c5060e ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:25:42.473Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:25:43.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4bb95e78-c298-43f0-a9b7-ed96ed0449e0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:25:43.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4bb95e78-c298-43f0-a9b7-ed96ed0449e0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f297581 2021-07-13T00:25:43.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4bb95e78-c298-43f0-a9b7-ed96ed0449e0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:25:43.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4bb95e78-c298-43f0-a9b7-ed96ed0449e0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:25:44.556Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=feb0532f-3e1e-4212-afc6-aab3c91f2d29 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:25:44.556Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=feb0532f-3e1e-4212-afc6-aab3c91f2d29 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@68be0d7b 2021-07-13T00:25:44.556Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=feb0532f-3e1e-4212-afc6-aab3c91f2d29 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:25:44.556Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=feb0532f-3e1e-4212-afc6-aab3c91f2d29 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:25:44.863Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=feb0532f-3e1e-4212-afc6-aab3c91f2d29 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:25:44.864Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=feb0532f-3e1e-4212-afc6-aab3c91f2d29 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:25:47.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a0edd32-564b-453e-bf47-c0afab5b47ea PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:25:47.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a0edd32-564b-453e-bf47-c0afab5b47ea PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4e0445f9 2021-07-13T00:25:47.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a0edd32-564b-453e-bf47-c0afab5b47ea PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:25:47.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a0edd32-564b-453e-bf47-c0afab5b47ea PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:25:50.770Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6053e4c-42d5-4d10-a237-3a40b1267217 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:25:50.770Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6053e4c-42d5-4d10-a237-3a40b1267217 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5945309d 2021-07-13T00:25:50.770Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6053e4c-42d5-4d10-a237-3a40b1267217 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:25:50.770Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6053e4c-42d5-4d10-a237-3a40b1267217 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:25:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:25:52.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:25:54.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0015e092-b62c-4122-a7db-979f7d571bb9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:25:54.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0015e092-b62c-4122-a7db-979f7d571bb9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6df4d349 2021-07-13T00:25:54.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0015e092-b62c-4122-a7db-979f7d571bb9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:25:54.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0015e092-b62c-4122-a7db-979f7d571bb9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:25:56.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc178233-ab7e-4da3-83bc-00bfd5213401 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:25:56.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc178233-ab7e-4da3-83bc-00bfd5213401 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7539ce59 2021-07-13T00:25:56.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc178233-ab7e-4da3-83bc-00bfd5213401 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:25:56.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cc178233-ab7e-4da3-83bc-00bfd5213401 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:25:59.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1fd8a4bc-9b8e-4c54-89f3-c7398b98b458 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:25:59.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1fd8a4bc-9b8e-4c54-89f3-c7398b98b458 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@19fe768a 2021-07-13T00:25:59.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1fd8a4bc-9b8e-4c54-89f3-c7398b98b458 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:25:59.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1fd8a4bc-9b8e-4c54-89f3-c7398b98b458 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:26:02.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1390742-bd6d-420e-91f3-a0b516ebdf88 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:26:02.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1390742-bd6d-420e-91f3-a0b516ebdf88 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@69cf6cf0 2021-07-13T00:26:02.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1390742-bd6d-420e-91f3-a0b516ebdf88 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:26:02.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1390742-bd6d-420e-91f3-a0b516ebdf88 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:26:05.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e380ed5-bd07-41f5-ad06-dc43c872b542 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:26:05.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e380ed5-bd07-41f5-ad06-dc43c872b542 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@22ea841b 2021-07-13T00:26:05.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e380ed5-bd07-41f5-ad06-dc43c872b542 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:26:05.969Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e380ed5-bd07-41f5-ad06-dc43c872b542 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:26:07.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:26:07.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:26:08.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ab57e6b-dc40-4e4f-b56f-d50bdbfe6b04 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:26:08.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ab57e6b-dc40-4e4f-b56f-d50bdbfe6b04 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@199add19 2021-07-13T00:26:08.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ab57e6b-dc40-4e4f-b56f-d50bdbfe6b04 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:26:08.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4ab57e6b-dc40-4e4f-b56f-d50bdbfe6b04 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:26:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:26:09.369Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:26:09.369Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:26:11.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98a67bb0-53fd-4ce1-b51a-f799cd6115fb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:26:11.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98a67bb0-53fd-4ce1-b51a-f799cd6115fb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@733aa286 2021-07-13T00:26:11.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98a67bb0-53fd-4ce1-b51a-f799cd6115fb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:26:11.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98a67bb0-53fd-4ce1-b51a-f799cd6115fb PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:26:14.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32eed418-d326-40f5-b5f5-f21de6d55920 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:26:14.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32eed418-d326-40f5-b5f5-f21de6d55920 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6100e8fd 2021-07-13T00:26:14.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32eed418-d326-40f5-b5f5-f21de6d55920 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:26:14.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32eed418-d326-40f5-b5f5-f21de6d55920 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:26:18.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5ad043d-e96b-457d-8938-918058b70985 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:26:18.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5ad043d-e96b-457d-8938-918058b70985 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@173c629a 2021-07-13T00:26:18.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5ad043d-e96b-457d-8938-918058b70985 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:26:18.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5ad043d-e96b-457d-8938-918058b70985 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:26:20.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=225410bd-9e3f-40ba-a982-505c724dde89 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:26:20.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=225410bd-9e3f-40ba-a982-505c724dde89 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@58010771 2021-07-13T00:26:20.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=225410bd-9e3f-40ba-a982-505c724dde89 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:26:20.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=225410bd-9e3f-40ba-a982-505c724dde89 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:26:22.918Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:26:22.927Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:26:23.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c7557ce-b615-4b89-94fa-897c255390a8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:26:23.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c7557ce-b615-4b89-94fa-897c255390a8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7e89b065 2021-07-13T00:26:23.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c7557ce-b615-4b89-94fa-897c255390a8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:26:23.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c7557ce-b615-4b89-94fa-897c255390a8 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:26:24.546Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:26:26.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9bc42e1a-3b6c-4d4c-b3e6-fab2668f4625 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:26:26.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9bc42e1a-3b6c-4d4c-b3e6-fab2668f4625 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1828e849 2021-07-13T00:26:26.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9bc42e1a-3b6c-4d4c-b3e6-fab2668f4625 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:26:26.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9bc42e1a-3b6c-4d4c-b3e6-fab2668f4625 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:26:28.302Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0dn0xuwboaue11ei6sn59u6pcl246 2021-07-13T00:26:28.302Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0spdesq82hx81w2zclynhn3yx252 2021-07-13T00:26:28.302Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01r5mo35puez5emg0c5dv6ext2244 2021-07-13T00:26:28.302Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01kwg6hbpwgoxz1d1zwdf22jlrr249 2021-07-13T00:26:28.302Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0dn0gj6gu1iipdnz30yyoolgf269 2021-07-13T00:26:28.302Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01s6jqrfkx72p3j3geyouhxu59271 2021-07-13T00:26:28.302Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0ovott6gqzi1aqox0jd1utld4265 2021-07-13T00:26:28.304Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0z18xiu9mxv02tpkpd6w6qpha253 2021-07-13T00:26:28.304Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01er8vc5w7es0d741lrquqt459272 2021-07-13T00:26:28.304Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01npe7e854ruq1ewbta53g7now255 2021-07-13T00:26:28.304Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01ec5d1q8vmwh1vtt2ni5rvy8i263 2021-07-13T00:26:28.304Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0ibk55xenrcqq1c32t5l9i5pru236 2021-07-13T00:26:28.304Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01ls6ccorvtv4a1harigv986x6s254 2021-07-13T00:26:28.304Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0j3z4owlkrbjt1fhwsnwf6c5wg248 2021-07-13T00:26:28.304Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0il3bwbpxrg1k1e7bktlarpzio243 2021-07-13T00:26:28.305Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01ft7r0nthzspw1mtnkxfkfybk1261 2021-07-13T00:26:28.305Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0wy8qnq8w64311ejn5agtan74q262 2021-07-13T00:26:28.305Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node08tbpdqa9x1le1w0sktwegyv0a256 2021-07-13T00:26:28.305Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0eoygzwekk92xokty3k9i6aeu270 2021-07-13T00:26:28.305Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node012l60umt394ob9vx563po1myx259 2021-07-13T00:26:28.305Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0m035cvwi789vi68ocenxfscb264 2021-07-13T00:26:28.305Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0mc1ug7jdbedoh04j8uhqdmh0238 2021-07-13T00:26:28.305Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node014q5g321zespa1a2jzjixu15au237 2021-07-13T00:26:28.305Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0qmq9enpg29y31hpomwqj1nmu5247 2021-07-13T00:26:28.305Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node015yo0bk18kw922g8c58zevy2r266 2021-07-13T00:26:28.305Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0cgu0cldtk8qd1aboe65ubodle250 2021-07-13T00:26:28.305Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0gxc6lxloihn81ikyy6ojf02m4251 2021-07-13T00:26:28.305Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01icr635r5hjbw8ujmy3slrhjv260 2021-07-13T00:26:28.306Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01mqlxhzaiimuhoqnu2eeetkig268 2021-07-13T00:26:28.306Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0pzs745ginf171xjhi6g4nn70o241 2021-07-13T00:26:28.306Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0pa8lf5c87dho1nfspd8fkt6qx240 2021-07-13T00:26:28.306Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0151y5pgcxir6t23micw51i38c239 2021-07-13T00:26:28.306Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0b39da7ku97nvh6jg6js28hv2242 2021-07-13T00:26:28.306Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01h3m7rq9a3dy9c5tweih0soye234 2021-07-13T00:26:28.306Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01l96b0mty78j1vxm9vwa68g7l245 2021-07-13T00:26:28.306Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0uosqnxprzqg41kaudg5cc33vc258 2021-07-13T00:26:28.306Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node017seczqah24dr1fvdm1wn1yizb257 2021-07-13T00:26:28.307Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0ef5ozzy55wfu11srpr9mtels3267 2021-07-13T00:26:29.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9bb88573-46a1-4025-8dd9-e7e3ba4e0da0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:26:29.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9bb88573-46a1-4025-8dd9-e7e3ba4e0da0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6cf47164 2021-07-13T00:26:29.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9bb88573-46a1-4025-8dd9-e7e3ba4e0da0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:26:29.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9bb88573-46a1-4025-8dd9-e7e3ba4e0da0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:26:32.776Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4813c3b0-518d-4890-b835-68e0e95e536c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:26:32.776Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4813c3b0-518d-4890-b835-68e0e95e536c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3c9ed904 2021-07-13T00:26:32.776Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4813c3b0-518d-4890-b835-68e0e95e536c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:26:32.777Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4813c3b0-518d-4890-b835-68e0e95e536c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:26:35.564Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd1ba03f-14b1-470f-9023-10eead1441df PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:26:35.564Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd1ba03f-14b1-470f-9023-10eead1441df PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@f275c6a 2021-07-13T00:26:35.564Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd1ba03f-14b1-470f-9023-10eead1441df PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:26:35.564Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd1ba03f-14b1-470f-9023-10eead1441df PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:26:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:26:37.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:26:38.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fddc346-c948-429b-9668-ee2eef85b76b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:26:38.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fddc346-c948-429b-9668-ee2eef85b76b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@50129767 2021-07-13T00:26:38.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fddc346-c948-429b-9668-ee2eef85b76b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:26:38.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fddc346-c948-429b-9668-ee2eef85b76b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:26:41.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1db32126-52c2-483d-84ff-63d6fda1fe91 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:26:41.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1db32126-52c2-483d-84ff-63d6fda1fe91 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f22b350 2021-07-13T00:26:41.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1db32126-52c2-483d-84ff-63d6fda1fe91 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:26:41.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1db32126-52c2-483d-84ff-63d6fda1fe91 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:26:44.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=29917e52-823c-48bc-b2e2-50a149aeaa31 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:26:44.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=29917e52-823c-48bc-b2e2-50a149aeaa31 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@553e1f49 2021-07-13T00:26:44.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=29917e52-823c-48bc-b2e2-50a149aeaa31 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:26:44.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=29917e52-823c-48bc-b2e2-50a149aeaa31 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:26:48.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=72b57121-5faa-48c9-a2cf-bfbbdfade2bf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:26:48.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=72b57121-5faa-48c9-a2cf-bfbbdfade2bf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@60cfdd88 2021-07-13T00:26:48.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=72b57121-5faa-48c9-a2cf-bfbbdfade2bf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:26:48.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=72b57121-5faa-48c9-a2cf-bfbbdfade2bf PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:26:50.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=23f797f4-9965-405a-926b-7af525f2fba9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:26:50.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=23f797f4-9965-405a-926b-7af525f2fba9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@24a2d759 2021-07-13T00:26:50.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=23f797f4-9965-405a-926b-7af525f2fba9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:26:50.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=23f797f4-9965-405a-926b-7af525f2fba9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:26:52.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:26:52.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:26:53.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00191e88-8dec-4bd7-8384-865850050967 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:26:53.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00191e88-8dec-4bd7-8384-865850050967 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2e1048ae 2021-07-13T00:26:53.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00191e88-8dec-4bd7-8384-865850050967 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:26:53.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00191e88-8dec-4bd7-8384-865850050967 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:26:56.783Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e47f530-4f03-4b4e-9256-dc316e14b3a1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:26:56.783Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e47f530-4f03-4b4e-9256-dc316e14b3a1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@b71ce72 2021-07-13T00:26:56.783Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e47f530-4f03-4b4e-9256-dc316e14b3a1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:26:56.783Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e47f530-4f03-4b4e-9256-dc316e14b3a1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:26:59.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fc12353-221f-4780-a2f9-0c4e4203d163 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:26:59.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fc12353-221f-4780-a2f9-0c4e4203d163 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@154c7109 2021-07-13T00:26:59.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fc12353-221f-4780-a2f9-0c4e4203d163 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:26:59.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fc12353-221f-4780-a2f9-0c4e4203d163 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:27:02.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6ce161d-ec75-4aac-96ab-8c5fc5582020 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:27:02.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6ce161d-ec75-4aac-96ab-8c5fc5582020 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@27e674e6 2021-07-13T00:27:02.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6ce161d-ec75-4aac-96ab-8c5fc5582020 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:27:02.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6ce161d-ec75-4aac-96ab-8c5fc5582020 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:27:05.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa016df2-9e91-41b7-8e0d-37d0761e75bb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:27:05.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa016df2-9e91-41b7-8e0d-37d0761e75bb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@b9f0d8c 2021-07-13T00:27:05.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa016df2-9e91-41b7-8e0d-37d0761e75bb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:27:05.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa016df2-9e91-41b7-8e0d-37d0761e75bb PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:27:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:27:07.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:27:08.576Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca3b3105-cf4e-4950-b0a9-cb34a04c28c9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:27:08.577Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca3b3105-cf4e-4950-b0a9-cb34a04c28c9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e1840d3 2021-07-13T00:27:08.577Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca3b3105-cf4e-4950-b0a9-cb34a04c28c9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:27:08.577Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca3b3105-cf4e-4950-b0a9-cb34a04c28c9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:27:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:27:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:27:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:27:11.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1774603f-3c12-4efb-8f6f-982b322393b1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:27:11.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1774603f-3c12-4efb-8f6f-982b322393b1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78aabc20 2021-07-13T00:27:11.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1774603f-3c12-4efb-8f6f-982b322393b1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:27:11.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1774603f-3c12-4efb-8f6f-982b322393b1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:27:14.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ac6e587-6fa6-4597-b425-4827a0c9ce8a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:27:14.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ac6e587-6fa6-4597-b425-4827a0c9ce8a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6c7fb697 2021-07-13T00:27:14.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ac6e587-6fa6-4597-b425-4827a0c9ce8a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:27:14.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ac6e587-6fa6-4597-b425-4827a0c9ce8a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:27:17.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92d200b3-20c4-4a09-9e97-c74d3d1a46d0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:27:17.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92d200b3-20c4-4a09-9e97-c74d3d1a46d0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1fff35f9 2021-07-13T00:27:17.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92d200b3-20c4-4a09-9e97-c74d3d1a46d0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:27:17.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92d200b3-20c4-4a09-9e97-c74d3d1a46d0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:27:20.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c57eb111-d9d7-4a11-8a38-d607c563d7b9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:27:20.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c57eb111-d9d7-4a11-8a38-d607c563d7b9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@532de00d 2021-07-13T00:27:20.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c57eb111-d9d7-4a11-8a38-d607c563d7b9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:27:20.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c57eb111-d9d7-4a11-8a38-d607c563d7b9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:27:22.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:27:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:27:24.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52990b1f-9f07-40e5-8228-9a741c7893b1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:27:24.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52990b1f-9f07-40e5-8228-9a741c7893b1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7e4b1df4 2021-07-13T00:27:24.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52990b1f-9f07-40e5-8228-9a741c7893b1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:27:24.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52990b1f-9f07-40e5-8228-9a741c7893b1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:27:24.361Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:27:26.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ab89d1b-cdce-47da-bc5c-11252d65d479 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:27:26.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ab89d1b-cdce-47da-bc5c-11252d65d479 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6f91d7a4 2021-07-13T00:27:26.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ab89d1b-cdce-47da-bc5c-11252d65d479 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:27:26.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ab89d1b-cdce-47da-bc5c-11252d65d479 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:27:29.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c62f2b2-c527-40fe-98df-b1ef89b2dad3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:27:29.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c62f2b2-c527-40fe-98df-b1ef89b2dad3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@9a77752 2021-07-13T00:27:29.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c62f2b2-c527-40fe-98df-b1ef89b2dad3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:27:29.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c62f2b2-c527-40fe-98df-b1ef89b2dad3 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:27:32.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4abb3e70-e1f6-4833-83a3-1081253c5707 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:27:32.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4abb3e70-e1f6-4833-83a3-1081253c5707 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@35b6d3e3 2021-07-13T00:27:32.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4abb3e70-e1f6-4833-83a3-1081253c5707 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:27:32.465Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4abb3e70-e1f6-4833-83a3-1081253c5707 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:27:35.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5937b56-897d-493a-80d5-522496b50c6b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:27:35.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5937b56-897d-493a-80d5-522496b50c6b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1334cc1c 2021-07-13T00:27:35.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5937b56-897d-493a-80d5-522496b50c6b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:27:35.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5937b56-897d-493a-80d5-522496b50c6b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:27:37.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:27:37.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:27:38.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92d3d37e-ee88-4230-923f-e2fc0cf68757 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:27:38.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92d3d37e-ee88-4230-923f-e2fc0cf68757 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@72f91529 2021-07-13T00:27:38.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92d3d37e-ee88-4230-923f-e2fc0cf68757 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:27:38.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92d3d37e-ee88-4230-923f-e2fc0cf68757 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:27:41.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34b604d1-a69e-47db-b269-3271346d84d2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:27:41.769Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34b604d1-a69e-47db-b269-3271346d84d2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@40d6f1c3 2021-07-13T00:27:41.769Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34b604d1-a69e-47db-b269-3271346d84d2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:27:41.769Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34b604d1-a69e-47db-b269-3271346d84d2 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:27:45.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c52225e-bef8-4b0b-b5e3-9d7afa1cb944 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:27:45.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c52225e-bef8-4b0b-b5e3-9d7afa1cb944 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@437c0bf1 2021-07-13T00:27:45.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c52225e-bef8-4b0b-b5e3-9d7afa1cb944 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:27:45.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c52225e-bef8-4b0b-b5e3-9d7afa1cb944 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:27:48.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9eb5d404-a7cc-4d2c-8307-35fca93bdd1e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:27:48.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9eb5d404-a7cc-4d2c-8307-35fca93bdd1e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@769d7bae 2021-07-13T00:27:48.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9eb5d404-a7cc-4d2c-8307-35fca93bdd1e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:27:48.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9eb5d404-a7cc-4d2c-8307-35fca93bdd1e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:27:50.471Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1eaba7af-3c90-4765-b9eb-8da8e99e7728 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:27:50.471Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1eaba7af-3c90-4765-b9eb-8da8e99e7728 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4cf76809 2021-07-13T00:27:50.471Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1eaba7af-3c90-4765-b9eb-8da8e99e7728 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:27:50.471Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1eaba7af-3c90-4765-b9eb-8da8e99e7728 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:27:52.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:27:52.925Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:27:53.479Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=1eaba7af-3c90-4765-b9eb-8da8e99e7728 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:27:53.480Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=1eaba7af-3c90-4765-b9eb-8da8e99e7728 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:27:53.480Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=1eaba7af-3c90-4765-b9eb-8da8e99e7728 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:27:53.481Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=1eaba7af-3c90-4765-b9eb-8da8e99e7728 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:27:54.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e2789d0-8be5-443d-b3ec-52a8f356044b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:27:54.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e2789d0-8be5-443d-b3ec-52a8f356044b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3bd8c0bd 2021-07-13T00:27:54.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e2789d0-8be5-443d-b3ec-52a8f356044b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:27:54.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e2789d0-8be5-443d-b3ec-52a8f356044b PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:27:55.166Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=1e2789d0-8be5-443d-b3ec-52a8f356044b ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:27:55.166Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=1e2789d0-8be5-443d-b3ec-52a8f356044b ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:27:56.562Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7383c866-7785-44de-9545-6b0f25b71de0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:27:56.562Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7383c866-7785-44de-9545-6b0f25b71de0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5053f1a 2021-07-13T00:27:56.562Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7383c866-7785-44de-9545-6b0f25b71de0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:27:56.562Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7383c866-7785-44de-9545-6b0f25b71de0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:27:59.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0655dc16-9058-4387-a3d2-dfc2b42b0635 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:27:59.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0655dc16-9058-4387-a3d2-dfc2b42b0635 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@72c5a893 2021-07-13T00:27:59.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0655dc16-9058-4387-a3d2-dfc2b42b0635 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:27:59.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0655dc16-9058-4387-a3d2-dfc2b42b0635 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:28:02.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b16dcfb-789b-4786-a0fc-582fc9d0e69f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:28:02.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b16dcfb-789b-4786-a0fc-582fc9d0e69f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@f65535e 2021-07-13T00:28:02.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b16dcfb-789b-4786-a0fc-582fc9d0e69f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:28:02.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8b16dcfb-789b-4786-a0fc-582fc9d0e69f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:28:05.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f9b8094-b3d6-4c22-b7a9-cc06bf92927a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:28:05.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f9b8094-b3d6-4c22-b7a9-cc06bf92927a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@545b60d1 2021-07-13T00:28:05.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f9b8094-b3d6-4c22-b7a9-cc06bf92927a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:28:05.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f9b8094-b3d6-4c22-b7a9-cc06bf92927a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:28:07.917Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:28:07.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:28:08.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b026bcc-047a-4bee-99a6-f1b5cae3e9b3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:28:08.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b026bcc-047a-4bee-99a6-f1b5cae3e9b3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@788a68af 2021-07-13T00:28:08.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b026bcc-047a-4bee-99a6-f1b5cae3e9b3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:28:08.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b026bcc-047a-4bee-99a6-f1b5cae3e9b3 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:28:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:28:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:28:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:28:11.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d41e0d37-a917-428f-98eb-224d9446b168 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:28:11.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d41e0d37-a917-428f-98eb-224d9446b168 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2754c61d 2021-07-13T00:28:11.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d41e0d37-a917-428f-98eb-224d9446b168 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:28:11.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d41e0d37-a917-428f-98eb-224d9446b168 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:28:14.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e154d812-d806-4cd9-903d-4fb52bb74bf7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:28:14.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e154d812-d806-4cd9-903d-4fb52bb74bf7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@28e8e0a 2021-07-13T00:28:14.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e154d812-d806-4cd9-903d-4fb52bb74bf7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:28:14.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e154d812-d806-4cd9-903d-4fb52bb74bf7 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:28:17.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01dac8b7-059b-4140-8cc9-c66fc5904ee8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:28:17.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01dac8b7-059b-4140-8cc9-c66fc5904ee8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@ddc2b62 2021-07-13T00:28:17.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01dac8b7-059b-4140-8cc9-c66fc5904ee8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:28:17.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01dac8b7-059b-4140-8cc9-c66fc5904ee8 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:28:21.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa68733c-babe-4561-979b-c0423df1029d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:28:21.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa68733c-babe-4561-979b-c0423df1029d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@26b8e566 2021-07-13T00:28:21.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa68733c-babe-4561-979b-c0423df1029d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:28:21.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa68733c-babe-4561-979b-c0423df1029d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:28:22.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:28:22.918Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:28:23.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c7756eb-6f2a-470a-b3b8-faa50c903586 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:28:23.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c7756eb-6f2a-470a-b3b8-faa50c903586 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@27ddab82 2021-07-13T00:28:23.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c7756eb-6f2a-470a-b3b8-faa50c903586 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:28:23.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c7756eb-6f2a-470a-b3b8-faa50c903586 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:28:25.798Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:28:26.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d31b3754-d0c1-4ee1-9739-6fcfef1f6394 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:28:26.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d31b3754-d0c1-4ee1-9739-6fcfef1f6394 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@51d58fad 2021-07-13T00:28:26.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d31b3754-d0c1-4ee1-9739-6fcfef1f6394 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:28:26.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d31b3754-d0c1-4ee1-9739-6fcfef1f6394 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:28:30.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c348b8f-9b5b-4998-9ca8-8ebbb7b09278 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:28:30.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c348b8f-9b5b-4998-9ca8-8ebbb7b09278 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3d3e29ce 2021-07-13T00:28:30.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c348b8f-9b5b-4998-9ca8-8ebbb7b09278 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:28:30.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c348b8f-9b5b-4998-9ca8-8ebbb7b09278 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:28:32.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5c1da4c-d257-419d-bb4d-95fe4e84a9a6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:28:32.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5c1da4c-d257-419d-bb4d-95fe4e84a9a6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5c4c78da 2021-07-13T00:28:32.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5c1da4c-d257-419d-bb4d-95fe4e84a9a6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:28:32.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b5c1da4c-d257-419d-bb4d-95fe4e84a9a6 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:28:35.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a1c487b-d501-4a3b-82a5-e90d32e395ef PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:28:35.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a1c487b-d501-4a3b-82a5-e90d32e395ef PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1607496a 2021-07-13T00:28:35.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a1c487b-d501-4a3b-82a5-e90d32e395ef PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:28:35.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a1c487b-d501-4a3b-82a5-e90d32e395ef PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:28:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:28:37.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:28:38.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7145b160-c814-45ff-8f2f-db85183c1e3c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:28:38.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7145b160-c814-45ff-8f2f-db85183c1e3c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6bd214e0 2021-07-13T00:28:38.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7145b160-c814-45ff-8f2f-db85183c1e3c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:28:38.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7145b160-c814-45ff-8f2f-db85183c1e3c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:28:41.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8484d3b1-7c25-4bdc-8986-f0dd6204b6f8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:28:41.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8484d3b1-7c25-4bdc-8986-f0dd6204b6f8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@27ee200d 2021-07-13T00:28:41.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8484d3b1-7c25-4bdc-8986-f0dd6204b6f8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:28:41.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8484d3b1-7c25-4bdc-8986-f0dd6204b6f8 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:28:44.376Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1487f7bf-2d5f-4a6f-90bb-3cc24a67d034 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:28:44.376Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1487f7bf-2d5f-4a6f-90bb-3cc24a67d034 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@185b1682 2021-07-13T00:28:44.376Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1487f7bf-2d5f-4a6f-90bb-3cc24a67d034 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:28:44.376Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1487f7bf-2d5f-4a6f-90bb-3cc24a67d034 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:28:48.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f09444c-ec59-48c8-bb9d-6e8684d8d298 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:28:48.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f09444c-ec59-48c8-bb9d-6e8684d8d298 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@34bb50fa 2021-07-13T00:28:48.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f09444c-ec59-48c8-bb9d-6e8684d8d298 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:28:48.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f09444c-ec59-48c8-bb9d-6e8684d8d298 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:28:50.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5cd7ab6-2cda-40a1-ba30-17284a3f78e7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:28:50.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5cd7ab6-2cda-40a1-ba30-17284a3f78e7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4fb53799 2021-07-13T00:28:50.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5cd7ab6-2cda-40a1-ba30-17284a3f78e7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:28:50.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5cd7ab6-2cda-40a1-ba30-17284a3f78e7 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:28:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:28:52.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:28:53.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca59f114-41c7-429a-8f74-afc7cff61c3d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:28:53.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca59f114-41c7-429a-8f74-afc7cff61c3d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@198d982a 2021-07-13T00:28:53.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca59f114-41c7-429a-8f74-afc7cff61c3d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:28:53.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca59f114-41c7-429a-8f74-afc7cff61c3d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:28:56.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0e758f4-383d-4df8-838e-5ca6d973dbc8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:28:56.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0e758f4-383d-4df8-838e-5ca6d973dbc8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5922629f 2021-07-13T00:28:56.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0e758f4-383d-4df8-838e-5ca6d973dbc8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:28:56.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0e758f4-383d-4df8-838e-5ca6d973dbc8 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:28:59.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ef5d8ab-2b8f-4345-8490-742b36e872b1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:28:59.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ef5d8ab-2b8f-4345-8490-742b36e872b1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@37f0c5c1 2021-07-13T00:28:59.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ef5d8ab-2b8f-4345-8490-742b36e872b1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:28:59.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ef5d8ab-2b8f-4345-8490-742b36e872b1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:29:03.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6d0addb-0c7c-462c-aaa6-c9bc42fed7b9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:29:03.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6d0addb-0c7c-462c-aaa6-c9bc42fed7b9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@511f3559 2021-07-13T00:29:03.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6d0addb-0c7c-462c-aaa6-c9bc42fed7b9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:29:03.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6d0addb-0c7c-462c-aaa6-c9bc42fed7b9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:29:05.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd2f7823-6f59-4987-9224-1b16195d1732 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:29:05.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd2f7823-6f59-4987-9224-1b16195d1732 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@24a6ab98 2021-07-13T00:29:05.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd2f7823-6f59-4987-9224-1b16195d1732 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:29:05.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd2f7823-6f59-4987-9224-1b16195d1732 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:29:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:29:07.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:29:08.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec7282be-f211-4870-928f-6fb2c9bd8187 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:29:08.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec7282be-f211-4870-928f-6fb2c9bd8187 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6e5a0309 2021-07-13T00:29:08.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec7282be-f211-4870-928f-6fb2c9bd8187 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:29:08.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec7282be-f211-4870-928f-6fb2c9bd8187 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:29:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:29:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:29:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:29:11.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f0fb923-4b0a-4e55-ae04-e73ae67f439a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:29:11.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f0fb923-4b0a-4e55-ae04-e73ae67f439a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f350dce 2021-07-13T00:29:11.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f0fb923-4b0a-4e55-ae04-e73ae67f439a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:29:11.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f0fb923-4b0a-4e55-ae04-e73ae67f439a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:29:15.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f2c50d4-7cf2-424d-a0aa-b49914f788a9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:29:15.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f2c50d4-7cf2-424d-a0aa-b49914f788a9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@77a84e8c 2021-07-13T00:29:15.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f2c50d4-7cf2-424d-a0aa-b49914f788a9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:29:15.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f2c50d4-7cf2-424d-a0aa-b49914f788a9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:29:17.672Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb561fc6-8b53-4d09-9ac1-60da6ef31804 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:29:17.672Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb561fc6-8b53-4d09-9ac1-60da6ef31804 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6643d18a 2021-07-13T00:29:17.672Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb561fc6-8b53-4d09-9ac1-60da6ef31804 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:29:17.672Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb561fc6-8b53-4d09-9ac1-60da6ef31804 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:29:21.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd027541-4afb-4eb8-8f89-ac51ed97b0ec PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:29:21.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd027541-4afb-4eb8-8f89-ac51ed97b0ec PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@583fd400 2021-07-13T00:29:21.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd027541-4afb-4eb8-8f89-ac51ed97b0ec PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:29:21.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd027541-4afb-4eb8-8f89-ac51ed97b0ec PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:29:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:29:22.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:29:24.456Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0712b3b0-910f-4566-abde-7f40607f304d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:29:24.456Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0712b3b0-910f-4566-abde-7f40607f304d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2f93f9b4 2021-07-13T00:29:24.456Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0712b3b0-910f-4566-abde-7f40607f304d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:29:24.456Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0712b3b0-910f-4566-abde-7f40607f304d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:29:26.384Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:29:27.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab69ba7b-2f4d-45fd-b603-efe8bf0ac00f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:29:27.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab69ba7b-2f4d-45fd-b603-efe8bf0ac00f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1a7a1ee4 2021-07-13T00:29:27.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab69ba7b-2f4d-45fd-b603-efe8bf0ac00f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:29:27.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab69ba7b-2f4d-45fd-b603-efe8bf0ac00f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:29:29.586Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa6b2c9e-3e38-40b2-b16a-6c433e808539 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:29:29.586Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa6b2c9e-3e38-40b2-b16a-6c433e808539 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@62cd43a0 2021-07-13T00:29:29.586Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa6b2c9e-3e38-40b2-b16a-6c433e808539 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:29:29.586Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa6b2c9e-3e38-40b2-b16a-6c433e808539 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:29:32.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c78445b-29dc-4b27-b58e-385516baf5e0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:29:32.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c78445b-29dc-4b27-b58e-385516baf5e0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@44f4142d 2021-07-13T00:29:32.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c78445b-29dc-4b27-b58e-385516baf5e0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:29:32.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c78445b-29dc-4b27-b58e-385516baf5e0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:29:35.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c541755-2fac-4c69-ba00-7d9aa93dad48 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:29:35.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c541755-2fac-4c69-ba00-7d9aa93dad48 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@749a1b5e 2021-07-13T00:29:35.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c541755-2fac-4c69-ba00-7d9aa93dad48 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:29:35.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c541755-2fac-4c69-ba00-7d9aa93dad48 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:29:37.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:29:37.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:29:39.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11bac978-488e-4dba-a41c-10f7f047b1fe PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:29:39.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11bac978-488e-4dba-a41c-10f7f047b1fe PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3e5da617 2021-07-13T00:29:39.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11bac978-488e-4dba-a41c-10f7f047b1fe PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:29:39.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11bac978-488e-4dba-a41c-10f7f047b1fe PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:29:41.273Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=081b476a-601b-4dc1-8ee4-7c2745b1f2cd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:29:41.273Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=081b476a-601b-4dc1-8ee4-7c2745b1f2cd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3605323f 2021-07-13T00:29:41.273Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=081b476a-601b-4dc1-8ee4-7c2745b1f2cd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:29:41.273Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=081b476a-601b-4dc1-8ee4-7c2745b1f2cd PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:29:44.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35059e02-442f-4ad4-9b9c-8d8df06f0a3d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:29:44.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35059e02-442f-4ad4-9b9c-8d8df06f0a3d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@52b40dae 2021-07-13T00:29:44.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35059e02-442f-4ad4-9b9c-8d8df06f0a3d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:29:44.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35059e02-442f-4ad4-9b9c-8d8df06f0a3d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:29:48.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d173e775-f2f4-4798-81eb-64d511bd39b9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:29:48.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d173e775-f2f4-4798-81eb-64d511bd39b9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7650ac4d 2021-07-13T00:29:48.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d173e775-f2f4-4798-81eb-64d511bd39b9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:29:48.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d173e775-f2f4-4798-81eb-64d511bd39b9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:29:50.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d3932f1-8222-497e-9cbc-bb5cd49369cf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:29:50.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d3932f1-8222-497e-9cbc-bb5cd49369cf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@25eec18e 2021-07-13T00:29:50.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d3932f1-8222-497e-9cbc-bb5cd49369cf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:29:50.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d3932f1-8222-497e-9cbc-bb5cd49369cf PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:29:52.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:29:52.929Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:29:53.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34aaf5d2-aafd-49a8-9f77-a37aab909b87 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:29:53.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34aaf5d2-aafd-49a8-9f77-a37aab909b87 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3ae59bc6 2021-07-13T00:29:53.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34aaf5d2-aafd-49a8-9f77-a37aab909b87 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:29:53.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34aaf5d2-aafd-49a8-9f77-a37aab909b87 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:29:56.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9a13bf3-b1bf-4e3c-9d65-5028ab4228d8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:29:56.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9a13bf3-b1bf-4e3c-9d65-5028ab4228d8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@48ce2849 2021-07-13T00:29:56.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9a13bf3-b1bf-4e3c-9d65-5028ab4228d8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:29:56.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c9a13bf3-b1bf-4e3c-9d65-5028ab4228d8 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:29:58.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:29:58.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:29:59.768Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=c9a13bf3-b1bf-4e3c-9d65-5028ab4228d8 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:29:59.770Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=c9a13bf3-b1bf-4e3c-9d65-5028ab4228d8 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:29:59.770Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c9a13bf3-b1bf-4e3c-9d65-5028ab4228d8 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:29:59.770Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c9a13bf3-b1bf-4e3c-9d65-5028ab4228d8 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:30:01.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d010472d-13e3-4b8b-a926-041226b66bdd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:30:01.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d010472d-13e3-4b8b-a926-041226b66bdd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3b8fd7d9 2021-07-13T00:30:01.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d010472d-13e3-4b8b-a926-041226b66bdd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:30:01.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d010472d-13e3-4b8b-a926-041226b66bdd PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:30:01.663Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:30:02.466Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=d010472d-13e3-4b8b-a926-041226b66bdd ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:30:02.466Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=d010472d-13e3-4b8b-a926-041226b66bdd ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:30:03.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18cb0c72-0051-43dc-b541-de2f07150c92 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:30:03.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18cb0c72-0051-43dc-b541-de2f07150c92 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7e931efd 2021-07-13T00:30:03.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18cb0c72-0051-43dc-b541-de2f07150c92 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:30:03.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18cb0c72-0051-43dc-b541-de2f07150c92 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:30:03.382Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=18cb0c72-0051-43dc-b541-de2f07150c92 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:30:03.382Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=18cb0c72-0051-43dc-b541-de2f07150c92 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:30:05.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15b18ca6-61fa-493f-833a-d5cb57fdfe57 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:30:05.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15b18ca6-61fa-493f-833a-d5cb57fdfe57 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4726ed06 2021-07-13T00:30:05.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15b18ca6-61fa-493f-833a-d5cb57fdfe57 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:30:05.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15b18ca6-61fa-493f-833a-d5cb57fdfe57 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:30:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:30:07.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:30:09.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42b95a0d-f892-4302-befe-c4d822ec45e0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:30:09.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42b95a0d-f892-4302-befe-c4d822ec45e0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@252db24 2021-07-13T00:30:09.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42b95a0d-f892-4302-befe-c4d822ec45e0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:30:09.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42b95a0d-f892-4302-befe-c4d822ec45e0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:30:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:30:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:30:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:30:12.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e92b2455-4426-4df8-a07f-95c3f0efb95c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:30:12.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e92b2455-4426-4df8-a07f-95c3f0efb95c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1eecd2c2 2021-07-13T00:30:12.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e92b2455-4426-4df8-a07f-95c3f0efb95c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:30:12.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e92b2455-4426-4df8-a07f-95c3f0efb95c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:30:14.478Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a101856-d234-4b4f-b63f-5abb4f2e8897 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:30:14.478Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a101856-d234-4b4f-b63f-5abb4f2e8897 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78ee4314 2021-07-13T00:30:14.478Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a101856-d234-4b4f-b63f-5abb4f2e8897 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:30:14.478Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a101856-d234-4b4f-b63f-5abb4f2e8897 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:30:17.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=768a5d04-d347-4e5a-a184-06646d59941d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:30:17.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=768a5d04-d347-4e5a-a184-06646d59941d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@850dfd2 2021-07-13T00:30:17.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=768a5d04-d347-4e5a-a184-06646d59941d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:30:17.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=768a5d04-d347-4e5a-a184-06646d59941d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:30:22.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=48591305-e949-486e-9411-3d2be6202018 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:30:22.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=48591305-e949-486e-9411-3d2be6202018 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1ccfbe07 2021-07-13T00:30:22.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=48591305-e949-486e-9411-3d2be6202018 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:30:22.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=48591305-e949-486e-9411-3d2be6202018 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:30:22.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:30:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:30:23.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2008a9e-ffb7-4295-b6d5-e3643afc1670 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:30:23.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2008a9e-ffb7-4295-b6d5-e3643afc1670 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64cd159e 2021-07-13T00:30:23.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2008a9e-ffb7-4295-b6d5-e3643afc1670 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:30:23.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2008a9e-ffb7-4295-b6d5-e3643afc1670 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:30:24.336Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:30:26.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a3fac81-02c3-4232-95e5-c2e180b0f240 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:30:26.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a3fac81-02c3-4232-95e5-c2e180b0f240 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6e311cf0 2021-07-13T00:30:26.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a3fac81-02c3-4232-95e5-c2e180b0f240 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:30:26.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5a3fac81-02c3-4232-95e5-c2e180b0f240 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:30:29.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b32dd79-4567-4e40-8973-704bd4e9e43f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:30:29.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b32dd79-4567-4e40-8973-704bd4e9e43f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6091e0ab 2021-07-13T00:30:29.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b32dd79-4567-4e40-8973-704bd4e9e43f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:30:29.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b32dd79-4567-4e40-8973-704bd4e9e43f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:30:32.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3df5a8ab-9af2-47b5-bfc1-3741ec3eb1c5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:30:32.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3df5a8ab-9af2-47b5-bfc1-3741ec3eb1c5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@14f4538 2021-07-13T00:30:32.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3df5a8ab-9af2-47b5-bfc1-3741ec3eb1c5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:30:32.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3df5a8ab-9af2-47b5-bfc1-3741ec3eb1c5 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:30:35.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4acb9f3d-401c-47de-bee6-18d8bde6b6e8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:30:35.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4acb9f3d-401c-47de-bee6-18d8bde6b6e8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@da2c16a 2021-07-13T00:30:35.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4acb9f3d-401c-47de-bee6-18d8bde6b6e8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:30:35.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4acb9f3d-401c-47de-bee6-18d8bde6b6e8 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:30:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:30:37.931Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:30:39.177Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a44f223-07ba-4a68-a5ab-155ae5469c6d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:30:39.177Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a44f223-07ba-4a68-a5ab-155ae5469c6d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6b129ab2 2021-07-13T00:30:39.177Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a44f223-07ba-4a68-a5ab-155ae5469c6d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:30:39.177Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1a44f223-07ba-4a68-a5ab-155ae5469c6d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:30:42.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70a2aa65-d177-4bbc-ab9f-7fccde313294 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:30:42.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70a2aa65-d177-4bbc-ab9f-7fccde313294 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4e9abc3 2021-07-13T00:30:42.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70a2aa65-d177-4bbc-ab9f-7fccde313294 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:30:42.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70a2aa65-d177-4bbc-ab9f-7fccde313294 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:30:45.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ac4505f-f6f9-4c4e-a7e6-9bfbff833f0e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:30:45.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ac4505f-f6f9-4c4e-a7e6-9bfbff833f0e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7ff458e3 2021-07-13T00:30:45.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ac4505f-f6f9-4c4e-a7e6-9bfbff833f0e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:30:45.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ac4505f-f6f9-4c4e-a7e6-9bfbff833f0e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:30:48.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13f102e1-39eb-42c0-baaf-7102cee2be30 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:30:48.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13f102e1-39eb-42c0-baaf-7102cee2be30 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64c67fe6 2021-07-13T00:30:48.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13f102e1-39eb-42c0-baaf-7102cee2be30 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:30:48.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13f102e1-39eb-42c0-baaf-7102cee2be30 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:30:51.666Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=13f102e1-39eb-42c0-baaf-7102cee2be30 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:30:51.666Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=13f102e1-39eb-42c0-baaf-7102cee2be30 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:30:51.667Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=13f102e1-39eb-42c0-baaf-7102cee2be30 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:30:51.667Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=13f102e1-39eb-42c0-baaf-7102cee2be30 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:30:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:30:52.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:30:52.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a448f94-511d-4b91-8fb6-786ec6b1c2b7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:30:52.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a448f94-511d-4b91-8fb6-786ec6b1c2b7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b7ebec6 2021-07-13T00:30:52.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a448f94-511d-4b91-8fb6-786ec6b1c2b7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:30:52.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a448f94-511d-4b91-8fb6-786ec6b1c2b7 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:30:53.377Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=8a448f94-511d-4b91-8fb6-786ec6b1c2b7 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:30:53.377Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=8a448f94-511d-4b91-8fb6-786ec6b1c2b7 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:30:54.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64b5e077-c2cd-4980-9cab-c7ad3ba0e6b0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:30:54.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64b5e077-c2cd-4980-9cab-c7ad3ba0e6b0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@f6ed03a 2021-07-13T00:30:54.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64b5e077-c2cd-4980-9cab-c7ad3ba0e6b0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:30:54.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64b5e077-c2cd-4980-9cab-c7ad3ba0e6b0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:30:57.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=509ac26a-b429-4c13-988e-0aafa59fae04 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:30:57.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=509ac26a-b429-4c13-988e-0aafa59fae04 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5800b77e 2021-07-13T00:30:57.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=509ac26a-b429-4c13-988e-0aafa59fae04 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:30:57.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=509ac26a-b429-4c13-988e-0aafa59fae04 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:30:59.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=93adebc6-b2ca-4176-9c74-f41423e78db5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:30:59.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=93adebc6-b2ca-4176-9c74-f41423e78db5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3666aa36 2021-07-13T00:30:59.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=93adebc6-b2ca-4176-9c74-f41423e78db5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:30:59.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=93adebc6-b2ca-4176-9c74-f41423e78db5 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:31:02.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4c69553-b0e2-44e2-97af-812ae27b2a55 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:31:02.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4c69553-b0e2-44e2-97af-812ae27b2a55 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@617f1352 2021-07-13T00:31:02.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4c69553-b0e2-44e2-97af-812ae27b2a55 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:31:02.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4c69553-b0e2-44e2-97af-812ae27b2a55 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:31:05.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c633189-e30e-4445-b225-5fd72fb8beda PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:31:05.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c633189-e30e-4445-b225-5fd72fb8beda PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1869e047 2021-07-13T00:31:05.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c633189-e30e-4445-b225-5fd72fb8beda PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:31:05.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c633189-e30e-4445-b225-5fd72fb8beda PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:31:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:31:07.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:31:08.795Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0437ce2b-cbf9-4d58-84c4-7a0bc1bbfa63 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:31:08.796Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0437ce2b-cbf9-4d58-84c4-7a0bc1bbfa63 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@450f1f73 2021-07-13T00:31:08.796Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0437ce2b-cbf9-4d58-84c4-7a0bc1bbfa63 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:31:08.796Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0437ce2b-cbf9-4d58-84c4-7a0bc1bbfa63 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:31:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:31:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:31:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:31:11.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb9df155-a72e-4c39-be73-bfa6f6921d01 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:31:11.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb9df155-a72e-4c39-be73-bfa6f6921d01 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@463b2afe 2021-07-13T00:31:11.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb9df155-a72e-4c39-be73-bfa6f6921d01 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:31:11.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb9df155-a72e-4c39-be73-bfa6f6921d01 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:31:14.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=910b173c-b374-45d4-8ff1-638adfab7fca PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:31:14.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=910b173c-b374-45d4-8ff1-638adfab7fca PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5fc754e8 2021-07-13T00:31:14.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=910b173c-b374-45d4-8ff1-638adfab7fca PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:31:14.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=910b173c-b374-45d4-8ff1-638adfab7fca PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:31:18.157Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ca72b6a-57b8-43af-91eb-857db587c856 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:31:18.157Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ca72b6a-57b8-43af-91eb-857db587c856 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@8fbfc8f 2021-07-13T00:31:18.157Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ca72b6a-57b8-43af-91eb-857db587c856 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:31:18.157Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ca72b6a-57b8-43af-91eb-857db587c856 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:31:20.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a50a74d-9887-4758-a3c2-6228a37e1c69 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:31:20.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a50a74d-9887-4758-a3c2-6228a37e1c69 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@35b29e64 2021-07-13T00:31:20.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a50a74d-9887-4758-a3c2-6228a37e1c69 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:31:20.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a50a74d-9887-4758-a3c2-6228a37e1c69 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:31:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:31:22.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:31:23.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24d92163-b983-445b-9c8a-909a014995db PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:31:23.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24d92163-b983-445b-9c8a-909a014995db PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ac12ef6 2021-07-13T00:31:23.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24d92163-b983-445b-9c8a-909a014995db PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:31:23.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=24d92163-b983-445b-9c8a-909a014995db PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:31:26.369Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:31:26.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1300c23-19ab-4167-83fd-af60f896a1b7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:31:26.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1300c23-19ab-4167-83fd-af60f896a1b7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4f445f52 2021-07-13T00:31:26.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1300c23-19ab-4167-83fd-af60f896a1b7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:31:26.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1300c23-19ab-4167-83fd-af60f896a1b7 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:31:30.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd1158f9-0874-4ad3-afdb-bb53e6ff548c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:31:30.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd1158f9-0874-4ad3-afdb-bb53e6ff548c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@180cddb1 2021-07-13T00:31:30.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd1158f9-0874-4ad3-afdb-bb53e6ff548c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:31:30.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd1158f9-0874-4ad3-afdb-bb53e6ff548c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:31:31.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:31:31.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:31:33.063Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=cd1158f9-0874-4ad3-afdb-bb53e6ff548c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:31:33.064Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=cd1158f9-0874-4ad3-afdb-bb53e6ff548c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:31:33.064Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=cd1158f9-0874-4ad3-afdb-bb53e6ff548c ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:31:33.064Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=cd1158f9-0874-4ad3-afdb-bb53e6ff548c ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:31:33.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0eff7de7-d173-45ec-b3f5-fe93dbbb111e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:31:33.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0eff7de7-d173-45ec-b3f5-fe93dbbb111e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@39ec316a 2021-07-13T00:31:33.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0eff7de7-d173-45ec-b3f5-fe93dbbb111e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:31:33.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0eff7de7-d173-45ec-b3f5-fe93dbbb111e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:31:34.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:31:37.010Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=0eff7de7-d173-45ec-b3f5-fe93dbbb111e ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:31:37.010Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=0eff7de7-d173-45ec-b3f5-fe93dbbb111e ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:31:37.010Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=0eff7de7-d173-45ec-b3f5-fe93dbbb111e ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:31:37.010Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=0eff7de7-d173-45ec-b3f5-fe93dbbb111e ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:31:37.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:31:37.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:31:37.926Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:31:38.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4d6ecdd-1b65-4524-8762-76bc417ea10a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:31:38.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4d6ecdd-1b65-4524-8762-76bc417ea10a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@772a6ced 2021-07-13T00:31:38.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4d6ecdd-1b65-4524-8762-76bc417ea10a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:31:38.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4d6ecdd-1b65-4524-8762-76bc417ea10a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:31:39.025Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f4d6ecdd-1b65-4524-8762-76bc417ea10a ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:31:39.025Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=f4d6ecdd-1b65-4524-8762-76bc417ea10a ErrorCategory=ERROR ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:31:39.812Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:31:40.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2950b92-fc19-4c58-a46a-1751e3946aee PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:31:40.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2950b92-fc19-4c58-a46a-1751e3946aee PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@510ac1c1 2021-07-13T00:31:40.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2950b92-fc19-4c58-a46a-1751e3946aee PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:31:40.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2950b92-fc19-4c58-a46a-1751e3946aee PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:31:41.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=894e3b77-3ec8-4c9e-b704-bb80f6972719 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:31:41.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=894e3b77-3ec8-4c9e-b704-bb80f6972719 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@350a539 2021-07-13T00:31:41.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=894e3b77-3ec8-4c9e-b704-bb80f6972719 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:31:41.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=894e3b77-3ec8-4c9e-b704-bb80f6972719 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:31:42.124Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=894e3b77-3ec8-4c9e-b704-bb80f6972719 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:31:42.124Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=894e3b77-3ec8-4c9e-b704-bb80f6972719 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:31:44.671Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=162b7064-690e-4892-b804-275dccc6f5f0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:31:44.671Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=162b7064-690e-4892-b804-275dccc6f5f0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@27b9fcba 2021-07-13T00:31:44.671Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=162b7064-690e-4892-b804-275dccc6f5f0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:31:44.671Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=162b7064-690e-4892-b804-275dccc6f5f0 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:31:48.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3d4af4b-4bc6-4768-b125-e24b784d6d39 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:31:48.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3d4af4b-4bc6-4768-b125-e24b784d6d39 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c5c07 2021-07-13T00:31:48.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3d4af4b-4bc6-4768-b125-e24b784d6d39 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:31:48.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3d4af4b-4bc6-4768-b125-e24b784d6d39 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:31:50.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=093e934b-7346-4da4-9fec-6fb352bc64ce PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:31:50.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=093e934b-7346-4da4-9fec-6fb352bc64ce PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@71f810c 2021-07-13T00:31:50.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=093e934b-7346-4da4-9fec-6fb352bc64ce PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:31:50.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=093e934b-7346-4da4-9fec-6fb352bc64ce PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:31:52.917Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:31:52.938Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:31:53.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aad3a077-902b-4505-8db7-fbfa189329a9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:31:53.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aad3a077-902b-4505-8db7-fbfa189329a9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@16085bb 2021-07-13T00:31:53.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aad3a077-902b-4505-8db7-fbfa189329a9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:31:53.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aad3a077-902b-4505-8db7-fbfa189329a9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:31:56.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=698ba0ce-af19-4ba9-8b7a-33c07abfd554 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:31:56.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=698ba0ce-af19-4ba9-8b7a-33c07abfd554 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@69d4e501 2021-07-13T00:31:56.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=698ba0ce-af19-4ba9-8b7a-33c07abfd554 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:31:56.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=698ba0ce-af19-4ba9-8b7a-33c07abfd554 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:31:59.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=891fbd4c-4513-40cf-8bf2-2a39dad6d1a7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:31:59.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=891fbd4c-4513-40cf-8bf2-2a39dad6d1a7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@b4a7058 2021-07-13T00:31:59.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=891fbd4c-4513-40cf-8bf2-2a39dad6d1a7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:31:59.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=891fbd4c-4513-40cf-8bf2-2a39dad6d1a7 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:32:02.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0bf7afb7-28c7-4792-b63b-52486c757bd9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:32:02.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0bf7afb7-28c7-4792-b63b-52486c757bd9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@702687dc 2021-07-13T00:32:02.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0bf7afb7-28c7-4792-b63b-52486c757bd9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:32:02.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0bf7afb7-28c7-4792-b63b-52486c757bd9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:32:04.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:32:04.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:32:05.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fbeb629f-1b6b-4650-90b1-e50ee45c96e9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:32:05.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fbeb629f-1b6b-4650-90b1-e50ee45c96e9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@75ee993e 2021-07-13T00:32:05.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fbeb629f-1b6b-4650-90b1-e50ee45c96e9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:32:05.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fbeb629f-1b6b-4650-90b1-e50ee45c96e9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:32:07.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:32:07.917Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:32:07.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:32:08.778Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=fbeb629f-1b6b-4650-90b1-e50ee45c96e9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:32:08.779Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=fbeb629f-1b6b-4650-90b1-e50ee45c96e9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:32:08.779Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=fbeb629f-1b6b-4650-90b1-e50ee45c96e9 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:32:08.779Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=fbeb629f-1b6b-4650-90b1-e50ee45c96e9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:32:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:32:09.369Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:32:09.369Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:32:09.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b2d3494-ab9b-4f1b-903a-d388042c27e1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:32:09.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b2d3494-ab9b-4f1b-903a-d388042c27e1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3bdc966f 2021-07-13T00:32:09.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b2d3494-ab9b-4f1b-903a-d388042c27e1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:32:09.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b2d3494-ab9b-4f1b-903a-d388042c27e1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:32:10.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:32:12.741Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=2b2d3494-ab9b-4f1b-903a-d388042c27e1 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:32:12.741Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=2b2d3494-ab9b-4f1b-903a-d388042c27e1 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:32:13.278Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f23e716f-1929-4212-98f8-4e20c233c5f3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:32:13.278Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f23e716f-1929-4212-98f8-4e20c233c5f3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@27109a2e 2021-07-13T00:32:13.278Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f23e716f-1929-4212-98f8-4e20c233c5f3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:32:13.278Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f23e716f-1929-4212-98f8-4e20c233c5f3 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:32:13.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:32:16.341Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=f23e716f-1929-4212-98f8-4e20c233c5f3 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:32:16.341Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=f23e716f-1929-4212-98f8-4e20c233c5f3 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:32:16.969Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98240c21-1bbc-4ab4-bfda-6b5cabaf43b4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:32:16.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98240c21-1bbc-4ab4-bfda-6b5cabaf43b4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@790645cc 2021-07-13T00:32:16.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98240c21-1bbc-4ab4-bfda-6b5cabaf43b4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:32:16.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98240c21-1bbc-4ab4-bfda-6b5cabaf43b4 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:32:20.022Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=98240c21-1bbc-4ab4-bfda-6b5cabaf43b4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:32:20.022Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=98240c21-1bbc-4ab4-bfda-6b5cabaf43b4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:32:20.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7704ad85-f186-4347-a5f0-e41151e8b319 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:32:20.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7704ad85-f186-4347-a5f0-e41151e8b319 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e63647d 2021-07-13T00:32:20.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7704ad85-f186-4347-a5f0-e41151e8b319 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:32:20.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7704ad85-f186-4347-a5f0-e41151e8b319 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:32:22.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:32:22.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:32:24.005Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=7704ad85-f186-4347-a5f0-e41151e8b319 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:32:24.005Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=7704ad85-f186-4347-a5f0-e41151e8b319 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:32:25.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a94e4e41-f624-4d00-9fb5-64107e44889a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:32:25.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a94e4e41-f624-4d00-9fb5-64107e44889a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5225103 2021-07-13T00:32:25.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a94e4e41-f624-4d00-9fb5-64107e44889a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:32:25.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a94e4e41-f624-4d00-9fb5-64107e44889a PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:32:26.161Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:32:28.600Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=a94e4e41-f624-4d00-9fb5-64107e44889a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:32:28.601Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=a94e4e41-f624-4d00-9fb5-64107e44889a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:32:29.071Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:32:29.525Z [ApplicationDataTypeCacheThread-1] INFO o.j.d.util.BackendOperation RequestID=null ErrorCategory=INFO ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=900 TargetEntity=ApplicationDataTypesCache Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Element.keys(Element.java:69) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getProperties(JanusGraphGenericDao.java:1491) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getByCriteria(JanusGraphGenericDao.java:759) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAllDataTypeNodes(PropertyOperation.java:1566) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.findAllDataTypesLazy(ApplicationDataTypeCache.java:309) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.hasDataTypesChanged(ApplicationDataTypeCache.java:220) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.run(ApplicationDataTypeCache.java:200) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:32:29.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fffc4ece-d9d7-4372-a875-4ef03fefffd3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:32:29.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fffc4ece-d9d7-4372-a875-4ef03fefffd3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@37d11a8d 2021-07-13T00:32:29.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fffc4ece-d9d7-4372-a875-4ef03fefffd3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:32:29.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fffc4ece-d9d7-4372-a875-4ef03fefffd3 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:32:32.423Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:32:32.423Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:32:32.901Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=fffc4ece-d9d7-4372-a875-4ef03fefffd3 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:32:32.901Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=fffc4ece-d9d7-4372-a875-4ef03fefffd3 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:32:32.901Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=fffc4ece-d9d7-4372-a875-4ef03fefffd3 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:32:32.901Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=fffc4ece-d9d7-4372-a875-4ef03fefffd3 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:32:33.423Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:32:34.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:32:36.121Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5e9b20c-888f-47f7-85a8-038953fb2374 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:32:36.121Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5e9b20c-888f-47f7-85a8-038953fb2374 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6440dc25 2021-07-13T00:32:36.121Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5e9b20c-888f-47f7-85a8-038953fb2374 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:32:36.121Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5e9b20c-888f-47f7-85a8-038953fb2374 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:32:36.762Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:32:36.864Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f5e9b20c-888f-47f7-85a8-038953fb2374 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:32:36.864Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=f5e9b20c-888f-47f7-85a8-038953fb2374 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:32:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:32:37.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:32:42.812Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6aa170b5-2795-44a2-8810-2e4d57b37621 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:32:42.813Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6aa170b5-2795-44a2-8810-2e4d57b37621 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@fbbcc1c 2021-07-13T00:32:42.813Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6aa170b5-2795-44a2-8810-2e4d57b37621 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:32:42.813Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6aa170b5-2795-44a2-8810-2e4d57b37621 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:32:43.419Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6aa170b5-2795-44a2-8810-2e4d57b37621 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:32:43.419Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6aa170b5-2795-44a2-8810-2e4d57b37621 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:32:44.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75d252a7-2682-474b-99ef-1ff27cbe0d0f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:32:44.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75d252a7-2682-474b-99ef-1ff27cbe0d0f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@15d489ca 2021-07-13T00:32:44.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75d252a7-2682-474b-99ef-1ff27cbe0d0f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:32:44.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75d252a7-2682-474b-99ef-1ff27cbe0d0f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:32:45.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87b19b10-e775-42f2-b63f-b857ac41c23e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:32:45.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87b19b10-e775-42f2-b63f-b857ac41c23e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7717cea1 2021-07-13T00:32:45.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87b19b10-e775-42f2-b63f-b857ac41c23e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:32:45.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=87b19b10-e775-42f2-b63f-b857ac41c23e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:32:47.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54a81f96-3d70-4d7a-a289-fdb998e02a3a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:32:47.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54a81f96-3d70-4d7a-a289-fdb998e02a3a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ac49b05 2021-07-13T00:32:47.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54a81f96-3d70-4d7a-a289-fdb998e02a3a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:32:47.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54a81f96-3d70-4d7a-a289-fdb998e02a3a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:32:49.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6db888a-e401-418f-8d04-30b357d7ef89 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:32:49.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6db888a-e401-418f-8d04-30b357d7ef89 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b4e756d 2021-07-13T00:32:49.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6db888a-e401-418f-8d04-30b357d7ef89 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:32:49.365Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6db888a-e401-418f-8d04-30b357d7ef89 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:32:51.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=722e8ed5-432e-472a-8bf1-c8a6fdfd9fbe PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:32:51.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=722e8ed5-432e-472a-8bf1-c8a6fdfd9fbe PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ae52517 2021-07-13T00:32:51.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=722e8ed5-432e-472a-8bf1-c8a6fdfd9fbe PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:32:51.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=722e8ed5-432e-472a-8bf1-c8a6fdfd9fbe PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:32:52.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a58a45c9-9f2c-4b56-81f8-277824e1a3cd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:32:52.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a58a45c9-9f2c-4b56-81f8-277824e1a3cd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63ce942c 2021-07-13T00:32:52.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a58a45c9-9f2c-4b56-81f8-277824e1a3cd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:32:52.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a58a45c9-9f2c-4b56-81f8-277824e1a3cd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:32:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:32:52.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:32:54.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1965252f-6ad1-45a5-9142-6c6d941dbd43 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:32:54.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1965252f-6ad1-45a5-9142-6c6d941dbd43 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@d3c27e3 2021-07-13T00:32:54.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1965252f-6ad1-45a5-9142-6c6d941dbd43 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:32:54.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1965252f-6ad1-45a5-9142-6c6d941dbd43 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:32:55.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d616e3ff-1a84-4631-8a29-1d2a1da0972f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:32:55.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d616e3ff-1a84-4631-8a29-1d2a1da0972f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6531de34 2021-07-13T00:32:55.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d616e3ff-1a84-4631-8a29-1d2a1da0972f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:32:55.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d616e3ff-1a84-4631-8a29-1d2a1da0972f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:32:57.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3340030e-9f96-4ac4-800f-1ab16d6a7121 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:32:57.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3340030e-9f96-4ac4-800f-1ab16d6a7121 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@667f9218 2021-07-13T00:32:57.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3340030e-9f96-4ac4-800f-1ab16d6a7121 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:32:57.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3340030e-9f96-4ac4-800f-1ab16d6a7121 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:33:00.072Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b41a8ba7-c2e6-40e7-99ce-d9ca803da034 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:33:00.072Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b41a8ba7-c2e6-40e7-99ce-d9ca803da034 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7187050f 2021-07-13T00:33:00.072Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b41a8ba7-c2e6-40e7-99ce-d9ca803da034 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:33:00.072Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b41a8ba7-c2e6-40e7-99ce-d9ca803da034 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:33:02.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9169f678-0eb0-4725-99b4-af5f5d018163 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:33:02.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9169f678-0eb0-4725-99b4-af5f5d018163 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1d6827e3 2021-07-13T00:33:02.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9169f678-0eb0-4725-99b4-af5f5d018163 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:33:02.871Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9169f678-0eb0-4725-99b4-af5f5d018163 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:33:05.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da6c85cd-9116-48cf-a7f2-ded39f108130 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:33:05.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da6c85cd-9116-48cf-a7f2-ded39f108130 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@76c39a04 2021-07-13T00:33:05.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da6c85cd-9116-48cf-a7f2-ded39f108130 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:33:05.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da6c85cd-9116-48cf-a7f2-ded39f108130 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:33:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:33:07.973Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:33:08.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9dd93ef2-0d13-46cb-8337-aba0d8213a4a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:33:08.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9dd93ef2-0d13-46cb-8337-aba0d8213a4a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6cb62996 2021-07-13T00:33:08.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9dd93ef2-0d13-46cb-8337-aba0d8213a4a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:33:08.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9dd93ef2-0d13-46cb-8337-aba0d8213a4a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:33:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:33:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:33:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:33:11.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b994d3fc-c82c-414e-8583-64754bb835ab PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:33:11.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b994d3fc-c82c-414e-8583-64754bb835ab PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@25d0e672 2021-07-13T00:33:11.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b994d3fc-c82c-414e-8583-64754bb835ab PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:33:11.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b994d3fc-c82c-414e-8583-64754bb835ab PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:33:15.316Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=baabc924-c6e8-4302-9915-99b43b682ca0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:33:15.316Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=baabc924-c6e8-4302-9915-99b43b682ca0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@79db8a14 2021-07-13T00:33:15.316Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=baabc924-c6e8-4302-9915-99b43b682ca0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:33:15.316Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=baabc924-c6e8-4302-9915-99b43b682ca0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:33:18.463Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b69618dd-c5ab-435b-acd8-94fac465154c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:33:18.464Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b69618dd-c5ab-435b-acd8-94fac465154c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@13a22c91 2021-07-13T00:33:18.464Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b69618dd-c5ab-435b-acd8-94fac465154c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:33:18.464Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b69618dd-c5ab-435b-acd8-94fac465154c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:33:20.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d64225bf-a71f-488c-b728-1eccc34578a8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:33:20.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d64225bf-a71f-488c-b728-1eccc34578a8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4e054dea 2021-07-13T00:33:20.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d64225bf-a71f-488c-b728-1eccc34578a8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:33:20.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d64225bf-a71f-488c-b728-1eccc34578a8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:33:22.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:33:22.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:33:23.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a03c6af-ffbd-44b4-919e-b732f967df8a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:33:23.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a03c6af-ffbd-44b4-919e-b732f967df8a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6c44d12e 2021-07-13T00:33:23.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a03c6af-ffbd-44b4-919e-b732f967df8a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:33:23.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a03c6af-ffbd-44b4-919e-b732f967df8a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:33:25.043Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:33:26.942Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d520df05-ea53-484b-bdc6-6f552f2aec2d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:33:26.942Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d520df05-ea53-484b-bdc6-6f552f2aec2d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@496a6d98 2021-07-13T00:33:26.942Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d520df05-ea53-484b-bdc6-6f552f2aec2d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:33:26.942Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d520df05-ea53-484b-bdc6-6f552f2aec2d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:33:29.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9cf41935-36cc-4320-91f7-287874caecd6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:33:29.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9cf41935-36cc-4320-91f7-287874caecd6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4bae0680 2021-07-13T00:33:29.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9cf41935-36cc-4320-91f7-287874caecd6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:33:29.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9cf41935-36cc-4320-91f7-287874caecd6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:33:32.810Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae61e847-3258-47b0-9746-32beeb2d73dd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:33:32.810Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae61e847-3258-47b0-9746-32beeb2d73dd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2fa07802 2021-07-13T00:33:32.810Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae61e847-3258-47b0-9746-32beeb2d73dd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:33:32.810Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae61e847-3258-47b0-9746-32beeb2d73dd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:33:35.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5eba771a-ecf7-4af2-abf8-8a246c2ef60d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:33:35.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5eba771a-ecf7-4af2-abf8-8a246c2ef60d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@65a71179 2021-07-13T00:33:35.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5eba771a-ecf7-4af2-abf8-8a246c2ef60d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:33:35.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5eba771a-ecf7-4af2-abf8-8a246c2ef60d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:33:37.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:33:37.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:33:38.463Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb8b20d4-536c-4d98-88a4-b6b8b2f1f2f2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:33:38.463Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb8b20d4-536c-4d98-88a4-b6b8b2f1f2f2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7fbba6f8 2021-07-13T00:33:38.463Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb8b20d4-536c-4d98-88a4-b6b8b2f1f2f2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:33:38.463Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb8b20d4-536c-4d98-88a4-b6b8b2f1f2f2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:33:42.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b0f15944-3cf9-4500-b49f-98edd2e0dd81 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:33:42.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b0f15944-3cf9-4500-b49f-98edd2e0dd81 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@441a2589 2021-07-13T00:33:42.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b0f15944-3cf9-4500-b49f-98edd2e0dd81 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:33:42.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b0f15944-3cf9-4500-b49f-98edd2e0dd81 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:33:44.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eea157b3-c2e3-4a5c-8cdd-4bb069b109ff PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:33:44.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eea157b3-c2e3-4a5c-8cdd-4bb069b109ff PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@68563560 2021-07-13T00:33:44.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eea157b3-c2e3-4a5c-8cdd-4bb069b109ff PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:33:44.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eea157b3-c2e3-4a5c-8cdd-4bb069b109ff PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:33:47.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81a93be0-b0b8-471a-b28a-ba7e8c106d00 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:33:47.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81a93be0-b0b8-471a-b28a-ba7e8c106d00 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@35a7ddca 2021-07-13T00:33:47.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81a93be0-b0b8-471a-b28a-ba7e8c106d00 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:33:47.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81a93be0-b0b8-471a-b28a-ba7e8c106d00 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:33:51.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=738a5ebd-1c54-4ae5-b8d8-7778724915ea PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:33:51.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=738a5ebd-1c54-4ae5-b8d8-7778724915ea PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@170931e0 2021-07-13T00:33:51.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=738a5ebd-1c54-4ae5-b8d8-7778724915ea PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:33:51.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=738a5ebd-1c54-4ae5-b8d8-7778724915ea PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:33:52.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:33:52.925Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:33:53.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=984e202e-1c55-4a5e-b58d-19111733d588 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:33:53.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=984e202e-1c55-4a5e-b58d-19111733d588 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@22a68e5 2021-07-13T00:33:53.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=984e202e-1c55-4a5e-b58d-19111733d588 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:33:53.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=984e202e-1c55-4a5e-b58d-19111733d588 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:33:56.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a706e430-1a31-4ace-b03e-ec11ee98267f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:33:56.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a706e430-1a31-4ace-b03e-ec11ee98267f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5b8ea0f5 2021-07-13T00:33:56.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a706e430-1a31-4ace-b03e-ec11ee98267f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:33:56.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a706e430-1a31-4ace-b03e-ec11ee98267f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:34:00.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31954bb1-bdb3-4bcc-9616-bdb7d97ca5b5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:34:00.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31954bb1-bdb3-4bcc-9616-bdb7d97ca5b5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@50cc6fc3 2021-07-13T00:34:00.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31954bb1-bdb3-4bcc-9616-bdb7d97ca5b5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:34:00.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31954bb1-bdb3-4bcc-9616-bdb7d97ca5b5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:34:02.971Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a445b91-566c-4f42-bdb4-b4a6c374e178 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:34:02.971Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a445b91-566c-4f42-bdb4-b4a6c374e178 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e14d20b 2021-07-13T00:34:02.971Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a445b91-566c-4f42-bdb4-b4a6c374e178 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:34:02.971Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a445b91-566c-4f42-bdb4-b4a6c374e178 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:34:05.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5928c995-f859-425a-a151-d3d77c363afd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:34:05.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5928c995-f859-425a-a151-d3d77c363afd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f28fe6c 2021-07-13T00:34:05.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5928c995-f859-425a-a151-d3d77c363afd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:34:05.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5928c995-f859-425a-a151-d3d77c363afd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:34:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:34:07.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:34:08.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ec32733-5b0c-4c64-aeed-0b951813bb30 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:34:08.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ec32733-5b0c-4c64-aeed-0b951813bb30 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@718f7474 2021-07-13T00:34:08.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ec32733-5b0c-4c64-aeed-0b951813bb30 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:34:08.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5ec32733-5b0c-4c64-aeed-0b951813bb30 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:34:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:34:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:34:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:34:11.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9283c6ad-780b-4762-b9e6-925f0ef7803e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:34:11.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9283c6ad-780b-4762-b9e6-925f0ef7803e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@420ebff1 2021-07-13T00:34:11.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9283c6ad-780b-4762-b9e6-925f0ef7803e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:34:11.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9283c6ad-780b-4762-b9e6-925f0ef7803e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:34:15.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=199c2f23-84f1-4323-85f8-319d7c96b975 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:34:15.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=199c2f23-84f1-4323-85f8-319d7c96b975 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6a22b931 2021-07-13T00:34:15.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=199c2f23-84f1-4323-85f8-319d7c96b975 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:34:15.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=199c2f23-84f1-4323-85f8-319d7c96b975 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:34:17.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f9e644e-63e4-48f5-9979-be11f1f78293 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:34:17.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f9e644e-63e4-48f5-9979-be11f1f78293 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@72e1e187 2021-07-13T00:34:17.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f9e644e-63e4-48f5-9979-be11f1f78293 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:34:17.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f9e644e-63e4-48f5-9979-be11f1f78293 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:34:20.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd13c8f6-37a7-48c5-b7a7-7829a6db91e5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:34:20.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd13c8f6-37a7-48c5-b7a7-7829a6db91e5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78c7077b 2021-07-13T00:34:20.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd13c8f6-37a7-48c5-b7a7-7829a6db91e5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:34:20.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd13c8f6-37a7-48c5-b7a7-7829a6db91e5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:34:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:34:22.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:34:23.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e9324ff2-21ba-496f-8d5d-552ea169b684 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:34:23.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e9324ff2-21ba-496f-8d5d-552ea169b684 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4f01dea5 2021-07-13T00:34:23.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e9324ff2-21ba-496f-8d5d-552ea169b684 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:34:23.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e9324ff2-21ba-496f-8d5d-552ea169b684 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:34:24.430Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:34:26.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8da5f130-473c-47e2-ad4d-27679fd6d4a7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:34:26.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8da5f130-473c-47e2-ad4d-27679fd6d4a7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@30151d45 2021-07-13T00:34:26.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8da5f130-473c-47e2-ad4d-27679fd6d4a7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:34:26.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8da5f130-473c-47e2-ad4d-27679fd6d4a7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:34:29.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e230b914-d1d3-45ff-9b00-21f13940e2c3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:34:29.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e230b914-d1d3-45ff-9b00-21f13940e2c3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4d052f19 2021-07-13T00:34:29.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e230b914-d1d3-45ff-9b00-21f13940e2c3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:34:29.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e230b914-d1d3-45ff-9b00-21f13940e2c3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:34:32.671Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9aee46b6-fcdd-490c-baca-78745e47155a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:34:32.671Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9aee46b6-fcdd-490c-baca-78745e47155a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@d39151d 2021-07-13T00:34:32.671Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9aee46b6-fcdd-490c-baca-78745e47155a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:34:32.672Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9aee46b6-fcdd-490c-baca-78745e47155a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:34:35.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03fac7c9-03e3-4ea0-9df7-e5424d8c9df7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:34:35.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03fac7c9-03e3-4ea0-9df7-e5424d8c9df7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4c0c16e9 2021-07-13T00:34:35.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03fac7c9-03e3-4ea0-9df7-e5424d8c9df7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:34:35.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03fac7c9-03e3-4ea0-9df7-e5424d8c9df7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:34:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:34:37.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:34:38.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd428fdc-7527-43c3-a7de-3a719d5f1088 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:34:38.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd428fdc-7527-43c3-a7de-3a719d5f1088 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@546bc518 2021-07-13T00:34:38.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd428fdc-7527-43c3-a7de-3a719d5f1088 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:34:38.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd428fdc-7527-43c3-a7de-3a719d5f1088 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:34:41.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e70699f-6620-4f64-b740-fc978aa12bb2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:34:41.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e70699f-6620-4f64-b740-fc978aa12bb2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@30f2556b 2021-07-13T00:34:41.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e70699f-6620-4f64-b740-fc978aa12bb2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:34:41.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e70699f-6620-4f64-b740-fc978aa12bb2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:34:44.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fbd2529-043b-4f69-a748-0575fd0d3250 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:34:44.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fbd2529-043b-4f69-a748-0575fd0d3250 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@75a62f95 2021-07-13T00:34:44.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fbd2529-043b-4f69-a748-0575fd0d3250 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:34:44.871Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3fbd2529-043b-4f69-a748-0575fd0d3250 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:34:48.567Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13a7b527-043b-4dfb-ba3f-7d024b7db01b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:34:48.567Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13a7b527-043b-4dfb-ba3f-7d024b7db01b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@59b1c496 2021-07-13T00:34:48.567Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13a7b527-043b-4dfb-ba3f-7d024b7db01b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:34:48.567Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13a7b527-043b-4dfb-ba3f-7d024b7db01b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:34:50.826Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5919ac29-dfcf-4a4c-b2cb-934e96ed470f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:34:50.826Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5919ac29-dfcf-4a4c-b2cb-934e96ed470f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@53f899bd 2021-07-13T00:34:50.826Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5919ac29-dfcf-4a4c-b2cb-934e96ed470f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:34:50.826Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5919ac29-dfcf-4a4c-b2cb-934e96ed470f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:34:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:34:52.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:34:54.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96d7b8df-2b3e-491a-b88f-1ef144f381bc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:34:54.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96d7b8df-2b3e-491a-b88f-1ef144f381bc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7183d7a4 2021-07-13T00:34:54.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96d7b8df-2b3e-491a-b88f-1ef144f381bc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:34:54.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96d7b8df-2b3e-491a-b88f-1ef144f381bc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:34:56.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7069b9d2-5521-40c1-8f8e-5e293073d4d4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:34:56.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7069b9d2-5521-40c1-8f8e-5e293073d4d4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@351d9486 2021-07-13T00:34:56.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7069b9d2-5521-40c1-8f8e-5e293073d4d4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:34:56.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7069b9d2-5521-40c1-8f8e-5e293073d4d4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:34:59.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=922920e2-2e20-444d-8af3-42967661fe92 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:34:59.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=922920e2-2e20-444d-8af3-42967661fe92 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@62df63c7 2021-07-13T00:34:59.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=922920e2-2e20-444d-8af3-42967661fe92 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:34:59.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=922920e2-2e20-444d-8af3-42967661fe92 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:35:02.362Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1768d534-d4c1-4b69-98b9-3d1d7baed939 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:35:02.362Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1768d534-d4c1-4b69-98b9-3d1d7baed939 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@75fd3b50 2021-07-13T00:35:02.363Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1768d534-d4c1-4b69-98b9-3d1d7baed939 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:35:02.363Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1768d534-d4c1-4b69-98b9-3d1d7baed939 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:35:05.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=534cacbd-3b76-4019-af91-e3a4c7c5ea69 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:35:05.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=534cacbd-3b76-4019-af91-e3a4c7c5ea69 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1a2cecbc 2021-07-13T00:35:05.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=534cacbd-3b76-4019-af91-e3a4c7c5ea69 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:35:05.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=534cacbd-3b76-4019-af91-e3a4c7c5ea69 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:35:07.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:35:07.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:35:08.781Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e569d7c-4778-40bc-b8aa-fa89bf2a16e9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:35:08.781Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e569d7c-4778-40bc-b8aa-fa89bf2a16e9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@43ccf2db 2021-07-13T00:35:08.781Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e569d7c-4778-40bc-b8aa-fa89bf2a16e9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:35:08.781Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e569d7c-4778-40bc-b8aa-fa89bf2a16e9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:35:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:35:09.369Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:35:09.369Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:35:11.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb656fe7-736e-4367-9d2e-95017a496c11 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:35:11.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb656fe7-736e-4367-9d2e-95017a496c11 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5a14603e 2021-07-13T00:35:11.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb656fe7-736e-4367-9d2e-95017a496c11 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:35:11.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb656fe7-736e-4367-9d2e-95017a496c11 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:35:13.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:35:13.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:35:14.675Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=fb656fe7-736e-4367-9d2e-95017a496c11 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:35:14.676Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=fb656fe7-736e-4367-9d2e-95017a496c11 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:35:14.676Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=fb656fe7-736e-4367-9d2e-95017a496c11 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:35:14.676Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=fb656fe7-736e-4367-9d2e-95017a496c11 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:35:16.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:35:17.721Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b4636202-8a3f-4fca-ae23-439f3d1cda34 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:35:17.721Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b4636202-8a3f-4fca-ae23-439f3d1cda34 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@462ff797 2021-07-13T00:35:17.721Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b4636202-8a3f-4fca-ae23-439f3d1cda34 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:35:17.722Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b4636202-8a3f-4fca-ae23-439f3d1cda34 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:35:18.719Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b4636202-8a3f-4fca-ae23-439f3d1cda34 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:35:18.719Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=b4636202-8a3f-4fca-ae23-439f3d1cda34 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:35:18.859Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:35:19.722Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c40cb7f-68b4-4d45-a0ac-e09a94b66f28 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:35:19.722Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c40cb7f-68b4-4d45-a0ac-e09a94b66f28 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f65cd70 2021-07-13T00:35:19.722Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c40cb7f-68b4-4d45-a0ac-e09a94b66f28 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:35:19.723Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c40cb7f-68b4-4d45-a0ac-e09a94b66f28 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:35:21.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=458702e3-659a-4de7-9c47-352211886d91 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:35:21.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=458702e3-659a-4de7-9c47-352211886d91 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@66894b32 2021-07-13T00:35:21.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=458702e3-659a-4de7-9c47-352211886d91 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:35:21.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=458702e3-659a-4de7-9c47-352211886d91 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:35:22.135Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=458702e3-659a-4de7-9c47-352211886d91 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:35:22.135Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=458702e3-659a-4de7-9c47-352211886d91 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:35:22.918Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:35:22.929Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:35:23.373Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a281266-83fd-4f36-9586-cff8f9e15777 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:35:23.373Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a281266-83fd-4f36-9586-cff8f9e15777 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f238b3b 2021-07-13T00:35:23.373Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a281266-83fd-4f36-9586-cff8f9e15777 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:35:23.373Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a281266-83fd-4f36-9586-cff8f9e15777 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:35:26.921Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9281721d-ed3c-44b9-983f-55cac0bc9148 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:35:26.921Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9281721d-ed3c-44b9-983f-55cac0bc9148 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3909f697 2021-07-13T00:35:26.921Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9281721d-ed3c-44b9-983f-55cac0bc9148 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:35:26.921Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9281721d-ed3c-44b9-983f-55cac0bc9148 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:35:27.007Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:35:30.271Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5b9a346-4dc8-4961-8b25-b49714be015e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:35:30.271Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5b9a346-4dc8-4961-8b25-b49714be015e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@836b28b 2021-07-13T00:35:30.271Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5b9a346-4dc8-4961-8b25-b49714be015e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:35:30.271Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5b9a346-4dc8-4961-8b25-b49714be015e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:35:32.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8fe36e41-682a-41fc-bb29-c28248f075c4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:35:32.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8fe36e41-682a-41fc-bb29-c28248f075c4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@244ab864 2021-07-13T00:35:32.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8fe36e41-682a-41fc-bb29-c28248f075c4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:35:32.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8fe36e41-682a-41fc-bb29-c28248f075c4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:35:36.317Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c94b6cef-af2c-4926-aea2-968787a1b4e7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:35:36.317Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c94b6cef-af2c-4926-aea2-968787a1b4e7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@67f38585 2021-07-13T00:35:36.317Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c94b6cef-af2c-4926-aea2-968787a1b4e7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:35:36.317Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c94b6cef-af2c-4926-aea2-968787a1b4e7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:35:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:35:37.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:35:38.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52822cfc-612c-420e-a004-80baf75d58e7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:35:38.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52822cfc-612c-420e-a004-80baf75d58e7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@11387aa2 2021-07-13T00:35:38.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52822cfc-612c-420e-a004-80baf75d58e7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:35:38.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52822cfc-612c-420e-a004-80baf75d58e7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:35:42.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c451ba5-856e-4cf1-8633-cb111c62c132 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:35:42.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c451ba5-856e-4cf1-8633-cb111c62c132 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2c2730e4 2021-07-13T00:35:42.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c451ba5-856e-4cf1-8633-cb111c62c132 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:35:42.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6c451ba5-856e-4cf1-8633-cb111c62c132 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:35:43.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:35:43.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:35:45.162Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=6c451ba5-856e-4cf1-8633-cb111c62c132 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:35:45.162Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=6c451ba5-856e-4cf1-8633-cb111c62c132 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:35:45.163Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6c451ba5-856e-4cf1-8633-cb111c62c132 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:35:45.163Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6c451ba5-856e-4cf1-8633-cb111c62c132 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:35:46.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:35:49.460Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:35:49.775Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73bf9c35-90f7-4ac6-a5da-d6762dc99757 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:35:49.775Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73bf9c35-90f7-4ac6-a5da-d6762dc99757 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5d5f349c 2021-07-13T00:35:49.775Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73bf9c35-90f7-4ac6-a5da-d6762dc99757 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:35:49.775Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73bf9c35-90f7-4ac6-a5da-d6762dc99757 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:35:50.260Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=73bf9c35-90f7-4ac6-a5da-d6762dc99757 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:35:50.260Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=73bf9c35-90f7-4ac6-a5da-d6762dc99757 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:35:51.138Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26f96ac5-41c9-4865-b9db-b897d901da46 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:35:51.138Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26f96ac5-41c9-4865-b9db-b897d901da46 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7444de18 2021-07-13T00:35:51.138Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26f96ac5-41c9-4865-b9db-b897d901da46 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:35:51.138Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26f96ac5-41c9-4865-b9db-b897d901da46 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:35:51.529Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=26f96ac5-41c9-4865-b9db-b897d901da46 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:35:51.530Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=26f96ac5-41c9-4865-b9db-b897d901da46 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:35:52.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:35:52.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:35:55.387Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b48e6c7-bebe-4a63-9570-da1f717cc6a1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:35:55.387Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b48e6c7-bebe-4a63-9570-da1f717cc6a1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@61716064 2021-07-13T00:35:55.387Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b48e6c7-bebe-4a63-9570-da1f717cc6a1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:35:55.387Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b48e6c7-bebe-4a63-9570-da1f717cc6a1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:35:56.714Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=83569d31-261b-43b1-aca6-e72d67086921 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:35:56.714Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=83569d31-261b-43b1-aca6-e72d67086921 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@452e5302 2021-07-13T00:35:56.714Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=83569d31-261b-43b1-aca6-e72d67086921 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:35:56.714Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=83569d31-261b-43b1-aca6-e72d67086921 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:35:58.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d570a8e-b84c-401a-beaf-7522a61112b5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:35:58.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d570a8e-b84c-401a-beaf-7522a61112b5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3941fd0d 2021-07-13T00:35:58.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d570a8e-b84c-401a-beaf-7522a61112b5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:35:58.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d570a8e-b84c-401a-beaf-7522a61112b5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:36:00.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42efc39f-7a29-44f2-900d-08960cd7ff0d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:36:00.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42efc39f-7a29-44f2-900d-08960cd7ff0d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6ba75023 2021-07-13T00:36:00.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42efc39f-7a29-44f2-900d-08960cd7ff0d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:36:00.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42efc39f-7a29-44f2-900d-08960cd7ff0d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:36:02.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec9044b2-dde6-431c-8e62-70d90b0f8ac1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:36:02.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec9044b2-dde6-431c-8e62-70d90b0f8ac1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1bf6f73 2021-07-13T00:36:02.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec9044b2-dde6-431c-8e62-70d90b0f8ac1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:36:02.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ec9044b2-dde6-431c-8e62-70d90b0f8ac1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:36:06.521Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be51069e-a1aa-45cd-b0a9-976b8abc27c7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:36:06.521Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be51069e-a1aa-45cd-b0a9-976b8abc27c7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@363e9a05 2021-07-13T00:36:06.521Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be51069e-a1aa-45cd-b0a9-976b8abc27c7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:36:06.521Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be51069e-a1aa-45cd-b0a9-976b8abc27c7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:36:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:36:07.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:36:09.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=238c7433-1945-4ada-97e9-d8ba2bd66132 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:36:09.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=238c7433-1945-4ada-97e9-d8ba2bd66132 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@a1a05a 2021-07-13T00:36:09.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=238c7433-1945-4ada-97e9-d8ba2bd66132 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:36:09.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=238c7433-1945-4ada-97e9-d8ba2bd66132 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:36:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:36:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:36:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:36:11.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55c7bb9e-c363-4fcc-9266-bb60c7eb78bd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:36:11.971Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55c7bb9e-c363-4fcc-9266-bb60c7eb78bd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@51d6ce25 2021-07-13T00:36:11.971Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55c7bb9e-c363-4fcc-9266-bb60c7eb78bd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:36:11.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55c7bb9e-c363-4fcc-9266-bb60c7eb78bd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:36:14.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a98fc802-9d96-439e-a64d-e663eb307735 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:36:14.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a98fc802-9d96-439e-a64d-e663eb307735 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2be8f79b 2021-07-13T00:36:14.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a98fc802-9d96-439e-a64d-e663eb307735 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:36:14.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a98fc802-9d96-439e-a64d-e663eb307735 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:36:17.972Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=a98fc802-9d96-439e-a64d-e663eb307735 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:36:17.973Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=a98fc802-9d96-439e-a64d-e663eb307735 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:36:17.973Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=a98fc802-9d96-439e-a64d-e663eb307735 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:36:17.973Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=a98fc802-9d96-439e-a64d-e663eb307735 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:36:19.123Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdd38d43-e106-4bf0-ace8-8577caf62cab PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:36:19.123Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdd38d43-e106-4bf0-ace8-8577caf62cab PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@69c4be51 2021-07-13T00:36:19.123Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdd38d43-e106-4bf0-ace8-8577caf62cab PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:36:19.123Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdd38d43-e106-4bf0-ace8-8577caf62cab PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:36:19.563Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=cdd38d43-e106-4bf0-ace8-8577caf62cab ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:36:19.564Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=cdd38d43-e106-4bf0-ace8-8577caf62cab ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:36:21.512Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0227765c-05ed-4d73-b95d-1aa21705190a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:36:21.512Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0227765c-05ed-4d73-b95d-1aa21705190a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@32a2d0d 2021-07-13T00:36:21.512Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0227765c-05ed-4d73-b95d-1aa21705190a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:36:21.512Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0227765c-05ed-4d73-b95d-1aa21705190a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:36:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:36:22.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:36:23.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71978719-4b93-4971-98c3-39c84091ff91 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:36:23.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71978719-4b93-4971-98c3-39c84091ff91 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6f9246dc 2021-07-13T00:36:23.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71978719-4b93-4971-98c3-39c84091ff91 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:36:23.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71978719-4b93-4971-98c3-39c84091ff91 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:36:25.758Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:36:27.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a22e53f-c006-4312-95d2-33655be32c61 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:36:27.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a22e53f-c006-4312-95d2-33655be32c61 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@23e84b49 2021-07-13T00:36:27.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a22e53f-c006-4312-95d2-33655be32c61 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:36:27.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a22e53f-c006-4312-95d2-33655be32c61 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:36:30.469Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=384595bf-0e5e-4ac6-8115-66b107c1abdf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:36:30.469Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=384595bf-0e5e-4ac6-8115-66b107c1abdf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@14b75f0f 2021-07-13T00:36:30.469Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=384595bf-0e5e-4ac6-8115-66b107c1abdf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:36:30.469Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=384595bf-0e5e-4ac6-8115-66b107c1abdf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:36:33.820Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae939128-2c3e-458f-bf7f-00a79b1165c6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:36:33.820Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae939128-2c3e-458f-bf7f-00a79b1165c6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@46275ad1 2021-07-13T00:36:33.820Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae939128-2c3e-458f-bf7f-00a79b1165c6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:36:33.820Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae939128-2c3e-458f-bf7f-00a79b1165c6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:36:36.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49fc692d-5893-4675-badd-8d3b463c02c9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:36:36.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49fc692d-5893-4675-badd-8d3b463c02c9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3b7e406f 2021-07-13T00:36:36.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49fc692d-5893-4675-badd-8d3b463c02c9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:36:36.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49fc692d-5893-4675-badd-8d3b463c02c9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:36:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:36:37.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:36:39.832Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b7d9cc3b-a670-4586-ba2d-0844d8ef145d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:36:39.833Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b7d9cc3b-a670-4586-ba2d-0844d8ef145d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5aadf64a 2021-07-13T00:36:39.833Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b7d9cc3b-a670-4586-ba2d-0844d8ef145d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:36:39.833Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b7d9cc3b-a670-4586-ba2d-0844d8ef145d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:36:41.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=39bb421b-43e9-4e84-a730-80ad63e37c41 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:36:41.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=39bb421b-43e9-4e84-a730-80ad63e37c41 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@744c9537 2021-07-13T00:36:41.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=39bb421b-43e9-4e84-a730-80ad63e37c41 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:36:41.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=39bb421b-43e9-4e84-a730-80ad63e37c41 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:36:44.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1324a6a8-30f9-4dd0-928a-f2ea0a88c5a1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:36:44.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1324a6a8-30f9-4dd0-928a-f2ea0a88c5a1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@841a8bd 2021-07-13T00:36:44.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1324a6a8-30f9-4dd0-928a-f2ea0a88c5a1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:36:44.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1324a6a8-30f9-4dd0-928a-f2ea0a88c5a1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:36:47.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a132473a-02e5-4d90-a396-35adcd160a1c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:36:47.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a132473a-02e5-4d90-a396-35adcd160a1c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@41f5ece2 2021-07-13T00:36:47.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a132473a-02e5-4d90-a396-35adcd160a1c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:36:47.857Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a132473a-02e5-4d90-a396-35adcd160a1c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:36:49.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:36:49.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:36:51.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6feacf85-fd04-4303-a36c-7f7af6b5e477 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:36:51.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6feacf85-fd04-4303-a36c-7f7af6b5e477 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@74df4d30 2021-07-13T00:36:51.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6feacf85-fd04-4303-a36c-7f7af6b5e477 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:36:51.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6feacf85-fd04-4303-a36c-7f7af6b5e477 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:36:51.617Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6feacf85-fd04-4303-a36c-7f7af6b5e477 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:36:51.617Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=6feacf85-fd04-4303-a36c-7f7af6b5e477 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:36:51.761Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:36:52.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:36:52.926Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:36:53.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e976b8d-ca9d-4568-9e2d-a1a4a6f765ac PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:36:53.857Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e976b8d-ca9d-4568-9e2d-a1a4a6f765ac PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64aa9bb1 2021-07-13T00:36:53.857Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e976b8d-ca9d-4568-9e2d-a1a4a6f765ac PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:36:53.857Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e976b8d-ca9d-4568-9e2d-a1a4a6f765ac PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:36:54.964Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=5e976b8d-ca9d-4568-9e2d-a1a4a6f765ac ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:36:54.964Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=5e976b8d-ca9d-4568-9e2d-a1a4a6f765ac ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:36:57.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a824c54-5104-49e2-977e-e46279e1722d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:36:57.157Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a824c54-5104-49e2-977e-e46279e1722d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@61abd3d9 2021-07-13T00:36:57.157Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a824c54-5104-49e2-977e-e46279e1722d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:36:57.157Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a824c54-5104-49e2-977e-e46279e1722d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:36:59.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=edca2d86-a52c-4df1-a121-352874f0e39c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:36:59.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=edca2d86-a52c-4df1-a121-352874f0e39c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@f57b1f2 2021-07-13T00:36:59.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=edca2d86-a52c-4df1-a121-352874f0e39c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:36:59.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=edca2d86-a52c-4df1-a121-352874f0e39c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:37:02.731Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d087cbed-da78-4e2a-bda1-5e48745e95ab PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:37:02.732Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d087cbed-da78-4e2a-bda1-5e48745e95ab PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1efcbe5 2021-07-13T00:37:02.732Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d087cbed-da78-4e2a-bda1-5e48745e95ab PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:37:02.732Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d087cbed-da78-4e2a-bda1-5e48745e95ab PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:37:05.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f658f80-8f1f-4e6d-a0c4-837ad04bcb46 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:37:05.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f658f80-8f1f-4e6d-a0c4-837ad04bcb46 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@31a8af7a 2021-07-13T00:37:05.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f658f80-8f1f-4e6d-a0c4-837ad04bcb46 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:37:05.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f658f80-8f1f-4e6d-a0c4-837ad04bcb46 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:37:07.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:37:07.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:37:08.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5f626c22-90ec-42b0-b494-f1770dca98ac PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:37:08.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5f626c22-90ec-42b0-b494-f1770dca98ac PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@ea2878d 2021-07-13T00:37:08.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5f626c22-90ec-42b0-b494-f1770dca98ac PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:37:08.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5f626c22-90ec-42b0-b494-f1770dca98ac PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:37:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:37:09.369Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:37:09.369Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:37:11.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a18df1d-4122-45a1-ae28-d00ed8fa474b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:37:11.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a18df1d-4122-45a1-ae28-d00ed8fa474b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3cc87f8a 2021-07-13T00:37:11.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a18df1d-4122-45a1-ae28-d00ed8fa474b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:37:11.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a18df1d-4122-45a1-ae28-d00ed8fa474b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:37:14.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b844392a-ce8b-4748-a119-579e92005f53 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:37:14.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b844392a-ce8b-4748-a119-579e92005f53 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@43073b15 2021-07-13T00:37:14.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b844392a-ce8b-4748-a119-579e92005f53 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:37:14.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b844392a-ce8b-4748-a119-579e92005f53 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:37:17.869Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=b844392a-ce8b-4748-a119-579e92005f53 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:37:17.870Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=b844392a-ce8b-4748-a119-579e92005f53 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:37:17.870Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b844392a-ce8b-4748-a119-579e92005f53 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:37:17.871Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b844392a-ce8b-4748-a119-579e92005f53 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:37:19.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:37:19.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:37:21.960Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:37:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:37:22.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:37:23.017Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab90c1dd-aa8c-4ae3-8a93-e9350be55114 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:37:23.017Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab90c1dd-aa8c-4ae3-8a93-e9350be55114 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@37a08f70 2021-07-13T00:37:23.018Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab90c1dd-aa8c-4ae3-8a93-e9350be55114 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:37:23.018Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ab90c1dd-aa8c-4ae3-8a93-e9350be55114 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:37:23.714Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=ab90c1dd-aa8c-4ae3-8a93-e9350be55114 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:37:23.715Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=ab90c1dd-aa8c-4ae3-8a93-e9350be55114 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:37:25.037Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:37:25.630Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6080f730-e0c5-499f-a38c-01a68b27faf3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:37:25.630Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6080f730-e0c5-499f-a38c-01a68b27faf3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@52efe6e7 2021-07-13T00:37:25.630Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6080f730-e0c5-499f-a38c-01a68b27faf3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:37:25.630Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6080f730-e0c5-499f-a38c-01a68b27faf3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:37:27.413Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=973f9c59-9a74-4e1c-bc18-1388005f91b0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:37:27.413Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=973f9c59-9a74-4e1c-bc18-1388005f91b0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@28caa03c 2021-07-13T00:37:27.413Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=973f9c59-9a74-4e1c-bc18-1388005f91b0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:37:27.413Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=973f9c59-9a74-4e1c-bc18-1388005f91b0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:37:28.308Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0faq622k98y9uz0h4qsirgwx2273 2021-07-13T00:37:28.308Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0nrnrk1ajqc4wasakd4f51dnr296 2021-07-13T00:37:28.308Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01iijedpp4rcoru2hc65bliod0291 2021-07-13T00:37:28.308Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0rqodv4xxhzjb1253j1hohl8zk299 2021-07-13T00:37:28.308Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0cv10au33oln51pqwrr460rkjz276 2021-07-13T00:37:28.308Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01iaizbbyt2mu615nkh6ztm274d289 2021-07-13T00:37:28.309Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0p6am1e9zzwtv1bnky29an1lyi280 2021-07-13T00:37:28.309Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node09537gdds6q5u18e1ywl3k9zbu283 2021-07-13T00:37:28.309Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01p75rl4jn0y5i1ced6yafr3m90297 2021-07-13T00:37:28.309Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0c2pg4n44om3c9v431tdex9ru281 2021-07-13T00:37:28.309Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01ak93rsr0aro2ft64yj9hnwmb300 2021-07-13T00:37:28.309Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node017crqf9olz5x0j3dtzco6taai287 2021-07-13T00:37:28.309Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0h9akhl3khd8uac6hpofd1qi3282 2021-07-13T00:37:28.309Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node018oavz7dle9sa1umc3ofiav923305 2021-07-13T00:37:28.309Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0at67lsgo57s91x3tx080hpgye277 2021-07-13T00:37:28.309Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01no9os9zfv21i1nt090phvtjin279 2021-07-13T00:37:28.309Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01kc2vtldx9am2f33c2rnwpda285 2021-07-13T00:37:28.309Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0k00qx9g1u24h7y2ysyths0hs278 2021-07-13T00:37:28.309Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0l7jygr2wx7k313jyeyfbbv2y3298 2021-07-13T00:37:28.309Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01dkwryv3veu031apdujdtdvm1n302 2021-07-13T00:37:28.309Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01svyn2jca3hfnhqjs7yg6ot5290 2021-07-13T00:37:28.309Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0hpxj4u5q6gfa1tn0si2kwwejg275 2021-07-13T00:37:28.310Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01wul1swfqssvi162qxxfyjfoa6294 2021-07-13T00:37:28.310Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01j5xqkxjinjur14gnbdx0qqmp1292 2021-07-13T00:37:28.310Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0g8nu90ijh5wiskhp7dwcgtdm304 2021-07-13T00:37:28.310Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01fmxr7q3dzffv14h2y4yqxbttr301 2021-07-13T00:37:28.310Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01r9e8rbjlikxuahonvucmrh4j286 2021-07-13T00:37:28.310Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0mw9zp0l22zd37btds3i6lvve295 2021-07-13T00:37:28.310Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0135bkm9ldv3hb1v8juzdq9bhsi288 2021-07-13T00:37:28.310Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0ohzq7h4dxzw61gw3hox9x6q03293 2021-07-13T00:37:28.310Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0m1y7pjy3sy7x7epfhzjr5qn5274 2021-07-13T00:37:28.310Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0lfvshygp2y6v1ka73scyz3ug5284 2021-07-13T00:37:31.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:37:31.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:37:31.818Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de6c23d0-1dd9-4d35-89ad-bd4eb2467dbd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:37:31.818Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de6c23d0-1dd9-4d35-89ad-bd4eb2467dbd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1ba2615a 2021-07-13T00:37:31.818Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de6c23d0-1dd9-4d35-89ad-bd4eb2467dbd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:37:31.818Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de6c23d0-1dd9-4d35-89ad-bd4eb2467dbd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:37:33.470Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:37:34.210Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=531acba9-cff2-45a7-864a-d3c4b034ce8a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:37:34.210Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=531acba9-cff2-45a7-864a-d3c4b034ce8a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@114bd910 2021-07-13T00:37:34.210Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=531acba9-cff2-45a7-864a-d3c4b034ce8a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:37:34.211Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=531acba9-cff2-45a7-864a-d3c4b034ce8a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:37:36.422Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=531acba9-cff2-45a7-864a-d3c4b034ce8a ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:37:36.422Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=531acba9-cff2-45a7-864a-d3c4b034ce8a ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:37:37.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c0804d7-b7bd-465c-b205-4cc822cd9456 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:37:37.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c0804d7-b7bd-465c-b205-4cc822cd9456 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3cbecd00 2021-07-13T00:37:37.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c0804d7-b7bd-465c-b205-4cc822cd9456 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:37:37.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c0804d7-b7bd-465c-b205-4cc822cd9456 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:37:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:37:37.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:37:38.365Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9c0804d7-b7bd-465c-b205-4cc822cd9456 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:37:38.365Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9c0804d7-b7bd-465c-b205-4cc822cd9456 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:37:40.154Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9bb0815e-41f5-42ca-b080-d46facf7cf2c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:37:40.154Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9bb0815e-41f5-42ca-b080-d46facf7cf2c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@418851eb 2021-07-13T00:37:40.154Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9bb0815e-41f5-42ca-b080-d46facf7cf2c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:37:40.154Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9bb0815e-41f5-42ca-b080-d46facf7cf2c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:37:42.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5dde421-7860-4a50-ae55-43c9107b22ee PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:37:42.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5dde421-7860-4a50-ae55-43c9107b22ee PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@49284031 2021-07-13T00:37:42.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5dde421-7860-4a50-ae55-43c9107b22ee PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:37:42.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5dde421-7860-4a50-ae55-43c9107b22ee PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:37:43.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb29d447-0046-491c-befa-6120a683aac5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:37:43.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb29d447-0046-491c-befa-6120a683aac5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@664935d 2021-07-13T00:37:43.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb29d447-0046-491c-befa-6120a683aac5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:37:43.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb29d447-0046-491c-befa-6120a683aac5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:37:44.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f46effa2-e390-4e43-9600-0308670ec07a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:37:44.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f46effa2-e390-4e43-9600-0308670ec07a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4cf8985c 2021-07-13T00:37:44.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f46effa2-e390-4e43-9600-0308670ec07a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:37:44.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f46effa2-e390-4e43-9600-0308670ec07a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:37:46.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:37:46.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:37:49.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:37:51.558Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:37:51.619Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=810c27aa-307f-4ca9-8fdf-cb0dc72b5e5a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:37:51.619Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=810c27aa-307f-4ca9-8fdf-cb0dc72b5e5a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ffc8f21 2021-07-13T00:37:51.619Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=810c27aa-307f-4ca9-8fdf-cb0dc72b5e5a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:37:51.619Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=810c27aa-307f-4ca9-8fdf-cb0dc72b5e5a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:37:51.967Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=810c27aa-307f-4ca9-8fdf-cb0dc72b5e5a ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:37:51.967Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=810c27aa-307f-4ca9-8fdf-cb0dc72b5e5a ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:37:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:37:52.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:37:52.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca551d97-fec2-490e-905c-58e3118d7686 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:37:52.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca551d97-fec2-490e-905c-58e3118d7686 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@44de7727 2021-07-13T00:37:52.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca551d97-fec2-490e-905c-58e3118d7686 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:37:52.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca551d97-fec2-490e-905c-58e3118d7686 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:37:53.665Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=ca551d97-fec2-490e-905c-58e3118d7686 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:37:53.665Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=ca551d97-fec2-490e-905c-58e3118d7686 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:37:54.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=58b19793-3ef5-4570-958e-1af10f984f87 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:37:54.560Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=58b19793-3ef5-4570-958e-1af10f984f87 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6682ea9a 2021-07-13T00:37:54.560Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=58b19793-3ef5-4570-958e-1af10f984f87 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:37:54.560Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=58b19793-3ef5-4570-958e-1af10f984f87 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:37:57.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a85bc3f3-4835-4eb5-a721-3cff1fffb5fd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:37:57.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a85bc3f3-4835-4eb5-a721-3cff1fffb5fd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6eb0a46e 2021-07-13T00:37:57.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a85bc3f3-4835-4eb5-a721-3cff1fffb5fd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:37:57.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a85bc3f3-4835-4eb5-a721-3cff1fffb5fd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:38:00.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=826d3556-cc95-4c1b-a78d-798162212728 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:38:00.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=826d3556-cc95-4c1b-a78d-798162212728 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@451eab1e 2021-07-13T00:38:00.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=826d3556-cc95-4c1b-a78d-798162212728 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:38:00.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=826d3556-cc95-4c1b-a78d-798162212728 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:38:02.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=112bb481-9325-402c-82c2-5654484c81d9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:38:02.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=112bb481-9325-402c-82c2-5654484c81d9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@724e8bd5 2021-07-13T00:38:02.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=112bb481-9325-402c-82c2-5654484c81d9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:38:02.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=112bb481-9325-402c-82c2-5654484c81d9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:38:04.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:38:04.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:38:05.866Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=112bb481-9325-402c-82c2-5654484c81d9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:38:05.866Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=112bb481-9325-402c-82c2-5654484c81d9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:38:05.867Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=112bb481-9325-402c-82c2-5654484c81d9 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:38:05.867Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=112bb481-9325-402c-82c2-5654484c81d9 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:38:06.618Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:38:07.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b2c92d3-8d2a-4026-b524-3ecf54c1750f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:38:07.264Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b2c92d3-8d2a-4026-b524-3ecf54c1750f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3e9693c 2021-07-13T00:38:07.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b2c92d3-8d2a-4026-b524-3ecf54c1750f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:38:07.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b2c92d3-8d2a-4026-b524-3ecf54c1750f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:38:07.771Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=5b2c92d3-8d2a-4026-b524-3ecf54c1750f ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:38:07.772Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=5b2c92d3-8d2a-4026-b524-3ecf54c1750f ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:38:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:38:07.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:38:08.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa279301-ddf8-4fde-b3dc-51efd9ed916d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:38:08.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa279301-ddf8-4fde-b3dc-51efd9ed916d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@58672eac 2021-07-13T00:38:08.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa279301-ddf8-4fde-b3dc-51efd9ed916d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:38:08.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa279301-ddf8-4fde-b3dc-51efd9ed916d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:38:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:38:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:38:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:38:09.560Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=fa279301-ddf8-4fde-b3dc-51efd9ed916d ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:38:09.561Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=fa279301-ddf8-4fde-b3dc-51efd9ed916d ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:38:11.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f082f66d-1bc4-4467-8ccf-c4cde63b0805 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:38:11.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f082f66d-1bc4-4467-8ccf-c4cde63b0805 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e711656 2021-07-13T00:38:11.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f082f66d-1bc4-4467-8ccf-c4cde63b0805 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:38:11.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f082f66d-1bc4-4467-8ccf-c4cde63b0805 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:38:14.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3aa487a6-1079-4c66-b98b-b5c425192ee0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:38:14.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3aa487a6-1079-4c66-b98b-b5c425192ee0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1699342d 2021-07-13T00:38:14.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3aa487a6-1079-4c66-b98b-b5c425192ee0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:38:14.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3aa487a6-1079-4c66-b98b-b5c425192ee0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:38:17.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c670ead2-5998-4fea-afc7-b2bed137aacf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:38:17.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c670ead2-5998-4fea-afc7-b2bed137aacf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@771e26f3 2021-07-13T00:38:17.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c670ead2-5998-4fea-afc7-b2bed137aacf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:38:17.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c670ead2-5998-4fea-afc7-b2bed137aacf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:38:20.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7769e543-cf7c-475d-8eae-8d3a10e55620 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:38:20.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7769e543-cf7c-475d-8eae-8d3a10e55620 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@166d74f2 2021-07-13T00:38:20.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7769e543-cf7c-475d-8eae-8d3a10e55620 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:38:20.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7769e543-cf7c-475d-8eae-8d3a10e55620 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:38:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:38:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:38:23.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad13390a-89eb-4b04-87db-97073ecfb7f0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:38:23.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad13390a-89eb-4b04-87db-97073ecfb7f0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@407813cf 2021-07-13T00:38:23.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad13390a-89eb-4b04-87db-97073ecfb7f0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:38:23.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad13390a-89eb-4b04-87db-97073ecfb7f0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:38:24.350Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:38:26.562Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd563d1e-02ba-417a-a5d3-88a504e86330 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:38:26.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd563d1e-02ba-417a-a5d3-88a504e86330 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f9f6fb1 2021-07-13T00:38:26.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd563d1e-02ba-417a-a5d3-88a504e86330 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:38:26.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fd563d1e-02ba-417a-a5d3-88a504e86330 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:38:29.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=400ecb31-4661-4281-80b8-cdfc20fec63a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:38:29.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=400ecb31-4661-4281-80b8-cdfc20fec63a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1930fb0f 2021-07-13T00:38:29.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=400ecb31-4661-4281-80b8-cdfc20fec63a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:38:29.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=400ecb31-4661-4281-80b8-cdfc20fec63a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:38:32.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=637de82d-ec02-4e95-8d79-2a6bc0a64d7d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:38:32.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=637de82d-ec02-4e95-8d79-2a6bc0a64d7d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6487c63f 2021-07-13T00:38:32.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=637de82d-ec02-4e95-8d79-2a6bc0a64d7d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:38:32.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=637de82d-ec02-4e95-8d79-2a6bc0a64d7d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:38:35.879Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c892d47-1c41-494d-80f4-7b2ec71911c0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:38:35.879Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c892d47-1c41-494d-80f4-7b2ec71911c0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6461570b 2021-07-13T00:38:35.879Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c892d47-1c41-494d-80f4-7b2ec71911c0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:38:35.879Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c892d47-1c41-494d-80f4-7b2ec71911c0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:38:37.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:38:37.927Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:38:39.267Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5169c617-f23e-45e4-8664-44aa64fb4994 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:38:39.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5169c617-f23e-45e4-8664-44aa64fb4994 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@674ed1 2021-07-13T00:38:39.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5169c617-f23e-45e4-8664-44aa64fb4994 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:38:39.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5169c617-f23e-45e4-8664-44aa64fb4994 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:38:42.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75f151e8-5966-4e24-9b89-d7e3aa94a4b6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:38:42.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75f151e8-5966-4e24-9b89-d7e3aa94a4b6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1e912551 2021-07-13T00:38:42.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75f151e8-5966-4e24-9b89-d7e3aa94a4b6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:38:42.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75f151e8-5966-4e24-9b89-d7e3aa94a4b6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:38:45.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81766880-28d3-48da-9e39-932ec47e1bab PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:38:45.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81766880-28d3-48da-9e39-932ec47e1bab PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@44fdae82 2021-07-13T00:38:45.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81766880-28d3-48da-9e39-932ec47e1bab PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:38:45.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81766880-28d3-48da-9e39-932ec47e1bab PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:38:47.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c8cad76-1380-4941-ba3d-1388c9811870 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:38:47.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c8cad76-1380-4941-ba3d-1388c9811870 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2aca6019 2021-07-13T00:38:47.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c8cad76-1380-4941-ba3d-1388c9811870 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:38:47.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c8cad76-1380-4941-ba3d-1388c9811870 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:38:51.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44d294fb-e314-4b56-9ab1-00be8f04e304 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:38:51.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44d294fb-e314-4b56-9ab1-00be8f04e304 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@10e1b257 2021-07-13T00:38:51.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44d294fb-e314-4b56-9ab1-00be8f04e304 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:38:51.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=44d294fb-e314-4b56-9ab1-00be8f04e304 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:38:52.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:38:52.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:38:57.131Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41179d08-8ee0-4322-a532-ee1afa20d9e5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:38:57.131Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41179d08-8ee0-4322-a532-ee1afa20d9e5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7cda6b7 2021-07-13T00:38:57.131Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41179d08-8ee0-4322-a532-ee1afa20d9e5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:38:57.131Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41179d08-8ee0-4322-a532-ee1afa20d9e5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:38:59.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b391f5df-d38e-44d2-aca4-45107a8d0b24 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:38:59.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b391f5df-d38e-44d2-aca4-45107a8d0b24 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2e6d8172 2021-07-13T00:38:59.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b391f5df-d38e-44d2-aca4-45107a8d0b24 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:38:59.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b391f5df-d38e-44d2-aca4-45107a8d0b24 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:39:00.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b4ac563-d4f5-4da2-b2ed-f4a450534c3b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:39:00.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b4ac563-d4f5-4da2-b2ed-f4a450534c3b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@57def60e 2021-07-13T00:39:00.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b4ac563-d4f5-4da2-b2ed-f4a450534c3b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:39:00.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b4ac563-d4f5-4da2-b2ed-f4a450534c3b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:39:02.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8baf2ea0-5ea5-456b-b8b8-747f72642d18 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:39:02.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8baf2ea0-5ea5-456b-b8b8-747f72642d18 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ab095a8 2021-07-13T00:39:02.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8baf2ea0-5ea5-456b-b8b8-747f72642d18 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:39:02.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8baf2ea0-5ea5-456b-b8b8-747f72642d18 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:39:05.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c279ac7-4085-4822-aab8-73932749d136 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:39:05.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c279ac7-4085-4822-aab8-73932749d136 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@20a26a57 2021-07-13T00:39:05.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c279ac7-4085-4822-aab8-73932749d136 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:39:05.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c279ac7-4085-4822-aab8-73932749d136 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:39:07.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:39:07.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:39:08.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4d7fedd-953b-4f57-9a76-a8706f6ce640 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:39:08.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4d7fedd-953b-4f57-9a76-a8706f6ce640 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@62061920 2021-07-13T00:39:08.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4d7fedd-953b-4f57-9a76-a8706f6ce640 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:39:08.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4d7fedd-953b-4f57-9a76-a8706f6ce640 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:39:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:39:09.369Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:39:09.369Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:39:11.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=77f5c4f8-6495-4897-9a10-1e73e48c1191 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:39:11.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=77f5c4f8-6495-4897-9a10-1e73e48c1191 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4248ede8 2021-07-13T00:39:11.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=77f5c4f8-6495-4897-9a10-1e73e48c1191 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:39:11.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=77f5c4f8-6495-4897-9a10-1e73e48c1191 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:39:14.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b0c854e6-2541-4f0c-918b-673a8717509d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:39:14.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b0c854e6-2541-4f0c-918b-673a8717509d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@c1b3ddc 2021-07-13T00:39:14.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b0c854e6-2541-4f0c-918b-673a8717509d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:39:14.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b0c854e6-2541-4f0c-918b-673a8717509d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:39:18.560Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7b2411a-c82b-44a0-91fc-91e8a7947caa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:39:18.560Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7b2411a-c82b-44a0-91fc-91e8a7947caa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3fd80eda 2021-07-13T00:39:18.560Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7b2411a-c82b-44a0-91fc-91e8a7947caa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:39:18.560Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7b2411a-c82b-44a0-91fc-91e8a7947caa PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:39:20.656Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52ec4192-791b-495d-a727-6c2647d99615 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:39:20.656Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52ec4192-791b-495d-a727-6c2647d99615 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6dc5cf18 2021-07-13T00:39:20.656Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52ec4192-791b-495d-a727-6c2647d99615 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:39:20.656Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52ec4192-791b-495d-a727-6c2647d99615 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:39:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:39:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:39:23.664Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=52ec4192-791b-495d-a727-6c2647d99615 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:39:23.665Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=52ec4192-791b-495d-a727-6c2647d99615 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:39:23.666Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=52ec4192-791b-495d-a727-6c2647d99615 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:39:23.666Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=52ec4192-791b-495d-a727-6c2647d99615 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:39:24.523Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54f4127f-53ec-4350-97c3-72bffce82813 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:39:24.523Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54f4127f-53ec-4350-97c3-72bffce82813 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@686748e5 2021-07-13T00:39:24.523Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54f4127f-53ec-4350-97c3-72bffce82813 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:39:24.523Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54f4127f-53ec-4350-97c3-72bffce82813 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:39:25.019Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=54f4127f-53ec-4350-97c3-72bffce82813 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:39:25.019Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=54f4127f-53ec-4350-97c3-72bffce82813 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:39:27.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=725f5ee4-4e7e-42d9-a9c3-b598419a8be0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:39:27.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=725f5ee4-4e7e-42d9-a9c3-b598419a8be0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@42161a54 2021-07-13T00:39:27.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=725f5ee4-4e7e-42d9-a9c3-b598419a8be0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:39:27.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=725f5ee4-4e7e-42d9-a9c3-b598419a8be0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:39:28.780Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:39:30.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94cbf816-9d6f-4e10-81a1-da4d39766a50 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:39:30.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94cbf816-9d6f-4e10-81a1-da4d39766a50 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1e8b001 2021-07-13T00:39:30.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94cbf816-9d6f-4e10-81a1-da4d39766a50 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:39:30.310Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94cbf816-9d6f-4e10-81a1-da4d39766a50 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:39:32.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3de98cd-07b6-44e3-8968-514631cddef8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:39:32.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3de98cd-07b6-44e3-8968-514631cddef8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@531ae0b9 2021-07-13T00:39:32.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3de98cd-07b6-44e3-8968-514631cddef8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:39:32.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3de98cd-07b6-44e3-8968-514631cddef8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:39:35.818Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f56b866-bc7c-408e-ba2d-6024ec9fcd17 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:39:35.818Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f56b866-bc7c-408e-ba2d-6024ec9fcd17 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@55b4ff0f 2021-07-13T00:39:35.818Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f56b866-bc7c-408e-ba2d-6024ec9fcd17 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:39:35.819Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f56b866-bc7c-408e-ba2d-6024ec9fcd17 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:39:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:39:37.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:39:39.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4530d8ae-9ee1-4e5b-a928-3c91d734ffc9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:39:39.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4530d8ae-9ee1-4e5b-a928-3c91d734ffc9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@27029b57 2021-07-13T00:39:39.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4530d8ae-9ee1-4e5b-a928-3c91d734ffc9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:39:39.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4530d8ae-9ee1-4e5b-a928-3c91d734ffc9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:39:41.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abac02c7-1324-45ef-9d22-846aafcd9c6c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:39:41.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abac02c7-1324-45ef-9d22-846aafcd9c6c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63ae5f0e 2021-07-13T00:39:41.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abac02c7-1324-45ef-9d22-846aafcd9c6c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:39:41.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abac02c7-1324-45ef-9d22-846aafcd9c6c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:39:43.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:39:43.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:39:45.269Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ccd68acb-454f-4ee9-b6ad-e8d4d27b278e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:39:45.269Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ccd68acb-454f-4ee9-b6ad-e8d4d27b278e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@673564ea 2021-07-13T00:39:45.269Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ccd68acb-454f-4ee9-b6ad-e8d4d27b278e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:39:45.269Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ccd68acb-454f-4ee9-b6ad-e8d4d27b278e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:39:45.663Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:39:45.674Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=ccd68acb-454f-4ee9-b6ad-e8d4d27b278e ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:39:45.674Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=ccd68acb-454f-4ee9-b6ad-e8d4d27b278e ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:39:47.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=392bf37d-2dda-4b41-b9a6-0d01415680c1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:39:47.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=392bf37d-2dda-4b41-b9a6-0d01415680c1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@150bb384 2021-07-13T00:39:47.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=392bf37d-2dda-4b41-b9a6-0d01415680c1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:39:47.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=392bf37d-2dda-4b41-b9a6-0d01415680c1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:39:48.164Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=392bf37d-2dda-4b41-b9a6-0d01415680c1 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:39:48.164Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=392bf37d-2dda-4b41-b9a6-0d01415680c1 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:39:51.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8d4a16a-a78c-40c3-87b1-12c822ea7132 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:39:51.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8d4a16a-a78c-40c3-87b1-12c822ea7132 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@26605a51 2021-07-13T00:39:51.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8d4a16a-a78c-40c3-87b1-12c822ea7132 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:39:51.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8d4a16a-a78c-40c3-87b1-12c822ea7132 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:39:52.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:39:52.926Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:39:54.072Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50bd0732-b109-4c79-8da3-fea11a667c1f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:39:54.072Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50bd0732-b109-4c79-8da3-fea11a667c1f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@372727bc 2021-07-13T00:39:54.072Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50bd0732-b109-4c79-8da3-fea11a667c1f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:39:54.072Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50bd0732-b109-4c79-8da3-fea11a667c1f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:39:56.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11044e76-4d4e-4575-a59d-2cf6d7df7413 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:39:56.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11044e76-4d4e-4575-a59d-2cf6d7df7413 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7b790786 2021-07-13T00:39:56.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11044e76-4d4e-4575-a59d-2cf6d7df7413 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:39:56.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11044e76-4d4e-4575-a59d-2cf6d7df7413 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:39:59.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0ee9f7c-9ec2-439f-a8c0-475a68b0620d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:39:59.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0ee9f7c-9ec2-439f-a8c0-475a68b0620d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@33260cfe 2021-07-13T00:39:59.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0ee9f7c-9ec2-439f-a8c0-475a68b0620d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:39:59.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0ee9f7c-9ec2-439f-a8c0-475a68b0620d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:40:03.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32d72091-9356-4f9e-906a-6445f38921e4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:40:03.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32d72091-9356-4f9e-906a-6445f38921e4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@17b07a19 2021-07-13T00:40:03.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32d72091-9356-4f9e-906a-6445f38921e4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:40:03.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32d72091-9356-4f9e-906a-6445f38921e4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:40:05.880Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88755c07-d11d-4ddb-a0bd-0c5599af88cf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:40:05.880Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88755c07-d11d-4ddb-a0bd-0c5599af88cf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@19a4b594 2021-07-13T00:40:05.880Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88755c07-d11d-4ddb-a0bd-0c5599af88cf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:40:05.880Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88755c07-d11d-4ddb-a0bd-0c5599af88cf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:40:07.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:40:07.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:40:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:40:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:40:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:40:09.823Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27309caa-aa3c-4cae-a01a-582cb1a0f4b8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:40:09.823Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27309caa-aa3c-4cae-a01a-582cb1a0f4b8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2a52d411 2021-07-13T00:40:09.823Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27309caa-aa3c-4cae-a01a-582cb1a0f4b8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:40:09.823Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=27309caa-aa3c-4cae-a01a-582cb1a0f4b8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:40:11.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fe277506-bf50-4d97-aa6c-9cc11181d1f0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:40:11.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fe277506-bf50-4d97-aa6c-9cc11181d1f0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@29906c59 2021-07-13T00:40:11.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fe277506-bf50-4d97-aa6c-9cc11181d1f0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:40:11.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fe277506-bf50-4d97-aa6c-9cc11181d1f0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:40:14.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fe51a2d7-e3a1-4fd9-b24e-9e5d7d793a0f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:40:14.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fe51a2d7-e3a1-4fd9-b24e-9e5d7d793a0f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3de4abc0 2021-07-13T00:40:14.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fe51a2d7-e3a1-4fd9-b24e-9e5d7d793a0f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:40:14.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fe51a2d7-e3a1-4fd9-b24e-9e5d7d793a0f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:40:18.214Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=82fd8070-c81b-49da-898d-5fadc5c30ff0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:40:18.214Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=82fd8070-c81b-49da-898d-5fadc5c30ff0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@150a1fd 2021-07-13T00:40:18.214Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=82fd8070-c81b-49da-898d-5fadc5c30ff0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:40:18.214Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=82fd8070-c81b-49da-898d-5fadc5c30ff0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:40:20.769Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e33fdf21-72bd-41f4-8dfe-7e6e8d99f663 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:40:20.769Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e33fdf21-72bd-41f4-8dfe-7e6e8d99f663 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3d1e0f9a 2021-07-13T00:40:20.769Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e33fdf21-72bd-41f4-8dfe-7e6e8d99f663 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:40:20.769Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e33fdf21-72bd-41f4-8dfe-7e6e8d99f663 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:40:22.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:40:22.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:40:23.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d5daf26-1776-4f9e-9ff8-c3a3da082399 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:40:23.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d5daf26-1776-4f9e-9ff8-c3a3da082399 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@17c64ec4 2021-07-13T00:40:23.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d5daf26-1776-4f9e-9ff8-c3a3da082399 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:40:23.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d5daf26-1776-4f9e-9ff8-c3a3da082399 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:40:24.499Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:40:26.926Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=369e9805-48d0-4eec-ae4e-8faa6ed230d3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:40:26.926Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=369e9805-48d0-4eec-ae4e-8faa6ed230d3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@fe44d57 2021-07-13T00:40:26.926Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=369e9805-48d0-4eec-ae4e-8faa6ed230d3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:40:26.926Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=369e9805-48d0-4eec-ae4e-8faa6ed230d3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:40:30.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=792544f7-004f-487c-87ea-33aa17d44635 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:40:30.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=792544f7-004f-487c-87ea-33aa17d44635 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@b3f5a1f 2021-07-13T00:40:30.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=792544f7-004f-487c-87ea-33aa17d44635 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:40:30.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=792544f7-004f-487c-87ea-33aa17d44635 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:40:32.646Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b6c3d62-fd37-4327-9a80-53eef83a46ba PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:40:32.646Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b6c3d62-fd37-4327-9a80-53eef83a46ba PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4610f63d 2021-07-13T00:40:32.646Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b6c3d62-fd37-4327-9a80-53eef83a46ba PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:40:32.646Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b6c3d62-fd37-4327-9a80-53eef83a46ba PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:40:36.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3edf1d1-0e01-47b1-be65-920f1f71aefe PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:40:36.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3edf1d1-0e01-47b1-be65-920f1f71aefe PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63053055 2021-07-13T00:40:36.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3edf1d1-0e01-47b1-be65-920f1f71aefe PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:40:36.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3edf1d1-0e01-47b1-be65-920f1f71aefe PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:40:37.917Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:40:37.926Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:40:38.577Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ccbebdc7-b4ca-4005-82f5-9eec56802abe PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:40:38.577Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ccbebdc7-b4ca-4005-82f5-9eec56802abe PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ea64797 2021-07-13T00:40:38.577Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ccbebdc7-b4ca-4005-82f5-9eec56802abe PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:40:38.578Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ccbebdc7-b4ca-4005-82f5-9eec56802abe PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:40:42.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be46a80c-15af-413a-b49c-5a8a799117fc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:40:42.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be46a80c-15af-413a-b49c-5a8a799117fc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3f3803fe 2021-07-13T00:40:42.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be46a80c-15af-413a-b49c-5a8a799117fc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:40:42.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be46a80c-15af-413a-b49c-5a8a799117fc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:40:45.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=938180d5-813a-4560-9997-007478c3b639 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:40:45.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=938180d5-813a-4560-9997-007478c3b639 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7ed8bb3a 2021-07-13T00:40:45.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=938180d5-813a-4560-9997-007478c3b639 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:40:45.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=938180d5-813a-4560-9997-007478c3b639 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:40:50.522Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50a7d1d8-4251-4422-9527-e992c7df09c8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:40:50.522Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50a7d1d8-4251-4422-9527-e992c7df09c8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3415ab2c 2021-07-13T00:40:50.522Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50a7d1d8-4251-4422-9527-e992c7df09c8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:40:50.522Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50a7d1d8-4251-4422-9527-e992c7df09c8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:40:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:40:52.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:40:53.529Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=50a7d1d8-4251-4422-9527-e992c7df09c8 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:40:53.530Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=50a7d1d8-4251-4422-9527-e992c7df09c8 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:40:53.531Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=50a7d1d8-4251-4422-9527-e992c7df09c8 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:40:53.531Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=50a7d1d8-4251-4422-9527-e992c7df09c8 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:41:04.560Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:41:04.560Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:41:05.561Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:41:06.559Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:41:07.612Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=80073423-fcee-48e5-9fa4-2a6eb95034d9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:41:07.612Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=80073423-fcee-48e5-9fa4-2a6eb95034d9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@48f3c8c6 2021-07-13T00:41:07.612Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=80073423-fcee-48e5-9fa4-2a6eb95034d9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:41:07.612Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=80073423-fcee-48e5-9fa4-2a6eb95034d9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:41:07.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:41:07.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:41:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:41:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:41:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:41:10.071Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=80073423-fcee-48e5-9fa4-2a6eb95034d9 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:41:10.071Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=80073423-fcee-48e5-9fa4-2a6eb95034d9 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:41:15.174Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af45827e-bbea-4fe9-b1b2-93611e12e62d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:41:15.174Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af45827e-bbea-4fe9-b1b2-93611e12e62d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@c916961 2021-07-13T00:41:15.174Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af45827e-bbea-4fe9-b1b2-93611e12e62d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:41:15.174Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=af45827e-bbea-4fe9-b1b2-93611e12e62d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:41:19.636Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8073c6da-e6e5-4028-9013-57ceb6e054c6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:41:19.636Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8073c6da-e6e5-4028-9013-57ceb6e054c6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1544ef14 2021-07-13T00:41:19.636Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8073c6da-e6e5-4028-9013-57ceb6e054c6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:41:19.636Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8073c6da-e6e5-4028-9013-57ceb6e054c6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:41:22.641Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=8073c6da-e6e5-4028-9013-57ceb6e054c6 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:41:22.641Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=8073c6da-e6e5-4028-9013-57ceb6e054c6 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:41:22.641Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=8073c6da-e6e5-4028-9013-57ceb6e054c6 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:41:22.641Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=8073c6da-e6e5-4028-9013-57ceb6e054c6 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:41:22.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:41:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:41:26.026Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79f9370e-9988-4783-bbf4-4a61eff44039 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:41:26.026Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79f9370e-9988-4783-bbf4-4a61eff44039 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4327b082 2021-07-13T00:41:26.026Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79f9370e-9988-4783-bbf4-4a61eff44039 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:41:26.026Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79f9370e-9988-4783-bbf4-4a61eff44039 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:41:26.770Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=79f9370e-9988-4783-bbf4-4a61eff44039 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:41:26.770Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=79f9370e-9988-4783-bbf4-4a61eff44039 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:41:29.024Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7c33a5e-8eca-48eb-a4f3-015757d44325 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:41:29.025Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7c33a5e-8eca-48eb-a4f3-015757d44325 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2b102e07 2021-07-13T00:41:29.025Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7c33a5e-8eca-48eb-a4f3-015757d44325 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:41:29.025Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7c33a5e-8eca-48eb-a4f3-015757d44325 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:41:32.726Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=603ac692-68c2-4181-a6e3-5895e5ac0088 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:41:32.726Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=603ac692-68c2-4181-a6e3-5895e5ac0088 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@199e583e 2021-07-13T00:41:32.726Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=603ac692-68c2-4181-a6e3-5895e5ac0088 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:41:32.726Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=603ac692-68c2-4181-a6e3-5895e5ac0088 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:41:34.754Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:41:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:41:37.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:41:40.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:41:40.460Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:41:43.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:41:46.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:41:47.437Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4318cf2f-0228-46ae-8406-aa5e36960b08 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:41:47.437Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4318cf2f-0228-46ae-8406-aa5e36960b08 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6743482f 2021-07-13T00:41:47.437Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4318cf2f-0228-46ae-8406-aa5e36960b08 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:41:47.437Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4318cf2f-0228-46ae-8406-aa5e36960b08 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:41:48.660Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:41:49.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2ef12e5-d6b6-4e2f-98a1-f31bd8ffb2f9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:41:49.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2ef12e5-d6b6-4e2f-98a1-f31bd8ffb2f9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@27caedb9 2021-07-13T00:41:49.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2ef12e5-d6b6-4e2f-98a1-f31bd8ffb2f9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:41:49.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e2ef12e5-d6b6-4e2f-98a1-f31bd8ffb2f9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:41:51.820Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=e2ef12e5-d6b6-4e2f-98a1-f31bd8ffb2f9 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:41:51.820Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=e2ef12e5-d6b6-4e2f-98a1-f31bd8ffb2f9 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:41:52.572Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66a367cd-2935-4e0e-96f9-9a23af321cec PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:41:52.572Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66a367cd-2935-4e0e-96f9-9a23af321cec PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@d009f51 2021-07-13T00:41:52.572Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66a367cd-2935-4e0e-96f9-9a23af321cec PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:41:52.572Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66a367cd-2935-4e0e-96f9-9a23af321cec PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:41:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:41:52.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:41:53.672Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=66a367cd-2935-4e0e-96f9-9a23af321cec ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:41:53.672Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=66a367cd-2935-4e0e-96f9-9a23af321cec ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:41:55.090Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=afdce885-e862-405c-a644-a5c08930aa53 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:41:55.090Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=afdce885-e862-405c-a644-a5c08930aa53 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@66347967 2021-07-13T00:41:55.090Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=afdce885-e862-405c-a644-a5c08930aa53 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:41:55.090Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=afdce885-e862-405c-a644-a5c08930aa53 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:41:56.463Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8b6872b-f25d-464c-ac1a-7a03528494ca PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:41:56.463Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8b6872b-f25d-464c-ac1a-7a03528494ca PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1bcb57be 2021-07-13T00:41:56.463Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8b6872b-f25d-464c-ac1a-7a03528494ca PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:41:56.463Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8b6872b-f25d-464c-ac1a-7a03528494ca PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:41:58.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f19910b8-e6b2-4eb0-b823-a1b0737098a9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:41:58.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f19910b8-e6b2-4eb0-b823-a1b0737098a9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6b1cd65b 2021-07-13T00:41:58.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f19910b8-e6b2-4eb0-b823-a1b0737098a9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:41:58.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f19910b8-e6b2-4eb0-b823-a1b0737098a9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:42:00.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f26e2521-6a3c-4c01-aad4-1647b1e63621 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:42:00.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f26e2521-6a3c-4c01-aad4-1647b1e63621 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@626f367e 2021-07-13T00:42:00.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f26e2521-6a3c-4c01-aad4-1647b1e63621 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:42:00.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f26e2521-6a3c-4c01-aad4-1647b1e63621 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:42:02.082Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6328106d-7ec9-491e-803b-68a4efef7f9b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:42:02.082Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6328106d-7ec9-491e-803b-68a4efef7f9b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@27d59415 2021-07-13T00:42:02.082Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6328106d-7ec9-491e-803b-68a4efef7f9b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:42:02.082Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6328106d-7ec9-491e-803b-68a4efef7f9b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:42:05.088Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=6328106d-7ec9-491e-803b-68a4efef7f9b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:42:05.088Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=6328106d-7ec9-491e-803b-68a4efef7f9b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:42:05.088Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6328106d-7ec9-491e-803b-68a4efef7f9b ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:42:05.088Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6328106d-7ec9-491e-803b-68a4efef7f9b ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:42:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:42:07.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:42:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:42:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:42:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:42:09.830Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e5628bd-89ff-4d8d-b7b9-18e2a7f30c57 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:42:09.830Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e5628bd-89ff-4d8d-b7b9-18e2a7f30c57 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@66e251f6 2021-07-13T00:42:09.830Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e5628bd-89ff-4d8d-b7b9-18e2a7f30c57 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:42:09.830Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e5628bd-89ff-4d8d-b7b9-18e2a7f30c57 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:42:12.881Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=3e5628bd-89ff-4d8d-b7b9-18e2a7f30c57 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:42:12.882Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=3e5628bd-89ff-4d8d-b7b9-18e2a7f30c57 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:42:22.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:42:22.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:42:22.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:42:22.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:42:24.329Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:42:24.814Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:42:25.727Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31b3b1a6-8cbf-4cd4-bef1-31f24f76c4b3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:42:25.727Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31b3b1a6-8cbf-4cd4-bef1-31f24f76c4b3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7086fabf 2021-07-13T00:42:25.727Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31b3b1a6-8cbf-4cd4-bef1-31f24f76c4b3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:42:25.727Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31b3b1a6-8cbf-4cd4-bef1-31f24f76c4b3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:42:27.563Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=31b3b1a6-8cbf-4cd4-bef1-31f24f76c4b3 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:42:27.563Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=31b3b1a6-8cbf-4cd4-bef1-31f24f76c4b3 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:42:28.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4c74f45-b90e-4a60-a4f7-21207141adb1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:42:28.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4c74f45-b90e-4a60-a4f7-21207141adb1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@f613eee 2021-07-13T00:42:28.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4c74f45-b90e-4a60-a4f7-21207141adb1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:42:28.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4c74f45-b90e-4a60-a4f7-21207141adb1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:42:31.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=550aa3d4-e0e7-4a27-8981-af29ffba4fa3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:42:31.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=550aa3d4-e0e7-4a27-8981-af29ffba4fa3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6baf8068 2021-07-13T00:42:31.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=550aa3d4-e0e7-4a27-8981-af29ffba4fa3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:42:31.017Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=550aa3d4-e0e7-4a27-8981-af29ffba4fa3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:42:32.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a52d6a2-ac3d-4cb7-ba6e-7aa96bb19112 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:42:32.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a52d6a2-ac3d-4cb7-ba6e-7aa96bb19112 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2c3bdca0 2021-07-13T00:42:32.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a52d6a2-ac3d-4cb7-ba6e-7aa96bb19112 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:42:32.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a52d6a2-ac3d-4cb7-ba6e-7aa96bb19112 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:42:35.129Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d70edc5-b742-47fa-bf44-b7a9dd5d8854 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:42:35.129Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d70edc5-b742-47fa-bf44-b7a9dd5d8854 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1291c03c 2021-07-13T00:42:35.129Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d70edc5-b742-47fa-bf44-b7a9dd5d8854 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:42:35.129Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d70edc5-b742-47fa-bf44-b7a9dd5d8854 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:42:36.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=30fbe3e9-04dc-47ed-bcdc-41087819f8ab PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:42:36.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=30fbe3e9-04dc-47ed-bcdc-41087819f8ab PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@14010ace 2021-07-13T00:42:36.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=30fbe3e9-04dc-47ed-bcdc-41087819f8ab PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:42:36.466Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=30fbe3e9-04dc-47ed-bcdc-41087819f8ab PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:42:37.917Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:42:37.926Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:42:38.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a79d079-f6dd-4bb8-ac44-a83557c0fce1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:42:38.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a79d079-f6dd-4bb8-ac44-a83557c0fce1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3ba74e6 2021-07-13T00:42:38.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a79d079-f6dd-4bb8-ac44-a83557c0fce1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:42:38.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a79d079-f6dd-4bb8-ac44-a83557c0fce1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:42:40.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a5789ee-df0f-4de6-b043-411a5d1331fd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:42:40.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a5789ee-df0f-4de6-b043-411a5d1331fd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@711666ed 2021-07-13T00:42:40.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a5789ee-df0f-4de6-b043-411a5d1331fd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:42:40.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3a5789ee-df0f-4de6-b043-411a5d1331fd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:42:41.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03b84b30-b6b9-40ec-bff5-8edbe29a2c40 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:42:41.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03b84b30-b6b9-40ec-bff5-8edbe29a2c40 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6d9ac8d3 2021-07-13T00:42:41.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03b84b30-b6b9-40ec-bff5-8edbe29a2c40 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:42:41.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03b84b30-b6b9-40ec-bff5-8edbe29a2c40 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:42:43.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41761a30-16e5-427a-96b8-a1b73849b21d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:42:43.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41761a30-16e5-427a-96b8-a1b73849b21d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7151f8d4 2021-07-13T00:42:43.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41761a30-16e5-427a-96b8-a1b73849b21d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:42:43.419Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41761a30-16e5-427a-96b8-a1b73849b21d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:42:44.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0745d711-4605-4057-ae7b-d3b44b126ce9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:42:44.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0745d711-4605-4057-ae7b-d3b44b126ce9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@211caf5c 2021-07-13T00:42:44.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0745d711-4605-4057-ae7b-d3b44b126ce9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:42:44.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0745d711-4605-4057-ae7b-d3b44b126ce9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:42:49.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=467eb20c-e964-45b2-a614-5ce536f78f90 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:42:49.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=467eb20c-e964-45b2-a614-5ce536f78f90 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@637f849c 2021-07-13T00:42:49.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=467eb20c-e964-45b2-a614-5ce536f78f90 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:42:49.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=467eb20c-e964-45b2-a614-5ce536f78f90 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:42:51.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89683a98-10a1-4efc-ba61-ff2fe4c8a009 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:42:51.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89683a98-10a1-4efc-ba61-ff2fe4c8a009 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@58980470 2021-07-13T00:42:51.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89683a98-10a1-4efc-ba61-ff2fe4c8a009 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:42:51.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=89683a98-10a1-4efc-ba61-ff2fe4c8a009 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:42:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:42:52.936Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:42:52.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13e110b4-772e-4ce4-b3f5-ccd09ed3cf54 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:42:52.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13e110b4-772e-4ce4-b3f5-ccd09ed3cf54 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@26f1c0ef 2021-07-13T00:42:52.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13e110b4-772e-4ce4-b3f5-ccd09ed3cf54 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:42:52.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13e110b4-772e-4ce4-b3f5-ccd09ed3cf54 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:42:54.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=223f7d5c-ab2c-4201-bdda-f5357bcf8649 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:42:54.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=223f7d5c-ab2c-4201-bdda-f5357bcf8649 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6253fd22 2021-07-13T00:42:54.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=223f7d5c-ab2c-4201-bdda-f5357bcf8649 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:42:54.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=223f7d5c-ab2c-4201-bdda-f5357bcf8649 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:42:56.175Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0fb7a9dd-eade-43e0-ac45-a2e218535f97 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:42:56.175Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0fb7a9dd-eade-43e0-ac45-a2e218535f97 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7d97ce90 2021-07-13T00:42:56.175Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0fb7a9dd-eade-43e0-ac45-a2e218535f97 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:42:56.175Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0fb7a9dd-eade-43e0-ac45-a2e218535f97 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:42:58.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=baed5563-420b-4914-ae23-53ee20eb4fdb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:42:58.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=baed5563-420b-4914-ae23-53ee20eb4fdb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@65b4eed6 2021-07-13T00:42:58.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=baed5563-420b-4914-ae23-53ee20eb4fdb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:42:58.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=baed5563-420b-4914-ae23-53ee20eb4fdb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:43:00.110Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ee06951-c554-45b7-9c69-9a06700d083b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:43:00.110Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ee06951-c554-45b7-9c69-9a06700d083b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4d6aac37 2021-07-13T00:43:00.110Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ee06951-c554-45b7-9c69-9a06700d083b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:43:00.110Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ee06951-c554-45b7-9c69-9a06700d083b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:43:01.363Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a0b9a452-ebc9-403a-b976-672f2694707b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:43:01.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a0b9a452-ebc9-403a-b976-672f2694707b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@302df255 2021-07-13T00:43:01.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a0b9a452-ebc9-403a-b976-672f2694707b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:43:01.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a0b9a452-ebc9-403a-b976-672f2694707b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:43:02.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff3fc57b-cd2c-4da6-99d8-c5be54ab7afe PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:43:02.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff3fc57b-cd2c-4da6-99d8-c5be54ab7afe PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@738d498e 2021-07-13T00:43:02.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff3fc57b-cd2c-4da6-99d8-c5be54ab7afe PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:43:02.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff3fc57b-cd2c-4da6-99d8-c5be54ab7afe PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:43:04.361Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c9f9cd9-8fd9-480c-b052-f92f01aa8935 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:43:04.361Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c9f9cd9-8fd9-480c-b052-f92f01aa8935 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@318dcb77 2021-07-13T00:43:04.361Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c9f9cd9-8fd9-480c-b052-f92f01aa8935 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:43:04.361Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c9f9cd9-8fd9-480c-b052-f92f01aa8935 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:43:06.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=094c6be1-3a79-444a-8493-20f9dc052478 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:43:06.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=094c6be1-3a79-444a-8493-20f9dc052478 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@299ed265 2021-07-13T00:43:06.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=094c6be1-3a79-444a-8493-20f9dc052478 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:43:06.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=094c6be1-3a79-444a-8493-20f9dc052478 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:43:07.560Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fdbe4ba3-78db-4e45-95cc-d31f024b16ae PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:43:07.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fdbe4ba3-78db-4e45-95cc-d31f024b16ae PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@76fe3259 2021-07-13T00:43:07.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fdbe4ba3-78db-4e45-95cc-d31f024b16ae PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:43:07.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fdbe4ba3-78db-4e45-95cc-d31f024b16ae PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:43:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:43:07.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:43:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:43:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:43:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:43:09.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ac3d272-903a-4551-900b-0af27c2f77b5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:43:09.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ac3d272-903a-4551-900b-0af27c2f77b5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6fa6d431 2021-07-13T00:43:09.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ac3d272-903a-4551-900b-0af27c2f77b5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:43:09.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ac3d272-903a-4551-900b-0af27c2f77b5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:43:12.563Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=7ac3d272-903a-4551-900b-0af27c2f77b5 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:43:12.564Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=7ac3d272-903a-4551-900b-0af27c2f77b5 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:43:12.564Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=7ac3d272-903a-4551-900b-0af27c2f77b5 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:43:12.565Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=7ac3d272-903a-4551-900b-0af27c2f77b5 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:43:22.762Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:43:22.762Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:43:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:43:22.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:43:23.763Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:43:24.336Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:43:24.763Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:43:25.763Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:43:26.764Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:43:28.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:43:31.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:43:32.667Z [cluster1-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.199.132:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.199.132:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:43:32.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c04d6996-9e98-4c85-9824-3aa157bb8fe5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:43:32.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c04d6996-9e98-4c85-9824-3aa157bb8fe5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@302f7cc1 2021-07-13T00:43:32.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c04d6996-9e98-4c85-9824-3aa157bb8fe5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:43:32.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c04d6996-9e98-4c85-9824-3aa157bb8fe5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:43:34.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:43:35.721Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=c04d6996-9e98-4c85-9824-3aa157bb8fe5 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:43:35.721Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=c04d6996-9e98-4c85-9824-3aa157bb8fe5 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:43:36.657Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a1cfbcd-46ca-4346-ac36-f785a2aebc09 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:43:36.657Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a1cfbcd-46ca-4346-ac36-f785a2aebc09 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@68960607 2021-07-13T00:43:36.657Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a1cfbcd-46ca-4346-ac36-f785a2aebc09 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:43:36.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4a1cfbcd-46ca-4346-ac36-f785a2aebc09 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:43:37.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:43:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:43:37.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:43:39.714Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=4a1cfbcd-46ca-4346-ac36-f785a2aebc09 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:43:39.714Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=4a1cfbcd-46ca-4346-ac36-f785a2aebc09 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:43:39.715Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=4a1cfbcd-46ca-4346-ac36-f785a2aebc09 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:43:39.715Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=4a1cfbcd-46ca-4346-ac36-f785a2aebc09 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:43:40.369Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0e2e8b9-2b84-4336-9238-9dee55341d41 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:43:40.370Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0e2e8b9-2b84-4336-9238-9dee55341d41 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7a4aebf4 2021-07-13T00:43:40.370Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0e2e8b9-2b84-4336-9238-9dee55341d41 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:43:40.370Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0e2e8b9-2b84-4336-9238-9dee55341d41 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:43:40.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:43:43.416Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=e0e2e8b9-2b84-4336-9238-9dee55341d41 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:43:43.416Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=e0e2e8b9-2b84-4336-9238-9dee55341d41 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:43:43.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:43:44.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a925af8f-02e1-4e1a-8033-ec105e53b9be PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:43:44.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a925af8f-02e1-4e1a-8033-ec105e53b9be PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2120b1b1 2021-07-13T00:43:44.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a925af8f-02e1-4e1a-8033-ec105e53b9be PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:43:44.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a925af8f-02e1-4e1a-8033-ec105e53b9be PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:43:46.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:43:49.162Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:43:49.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a078a4ef-afd4-4b1b-98cd-273651319748 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:43:49.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a078a4ef-afd4-4b1b-98cd-273651319748 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7fa0ba91 2021-07-13T00:43:49.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a078a4ef-afd4-4b1b-98cd-273651319748 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:43:49.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a078a4ef-afd4-4b1b-98cd-273651319748 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:43:50.222Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=a078a4ef-afd4-4b1b-98cd-273651319748 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:43:50.222Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=a078a4ef-afd4-4b1b-98cd-273651319748 ErrorCategory=ERROR ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:43:51.716Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36441ecb-373f-4161-ac6b-519929db45dc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:43:51.716Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36441ecb-373f-4161-ac6b-519929db45dc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@77c36078 2021-07-13T00:43:51.716Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36441ecb-373f-4161-ac6b-519929db45dc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:43:51.716Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36441ecb-373f-4161-ac6b-519929db45dc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:43:52.263Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=36441ecb-373f-4161-ac6b-519929db45dc ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:43:52.263Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=36441ecb-373f-4161-ac6b-519929db45dc ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:43:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:43:52.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:43:54.117Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95d354c2-633b-4f56-ae8d-3bc648411a9b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:43:54.117Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95d354c2-633b-4f56-ae8d-3bc648411a9b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@53caea08 2021-07-13T00:43:54.117Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95d354c2-633b-4f56-ae8d-3bc648411a9b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:43:54.117Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=95d354c2-633b-4f56-ae8d-3bc648411a9b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:43:55.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:43:55.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:43:56.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=239d356b-3ce4-4aab-8164-5b8ced7e9175 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:43:56.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=239d356b-3ce4-4aab-8164-5b8ced7e9175 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@46340d9a 2021-07-13T00:43:56.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=239d356b-3ce4-4aab-8164-5b8ced7e9175 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:43:56.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=239d356b-3ce4-4aab-8164-5b8ced7e9175 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:43:57.468Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:43:57.628Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=239d356b-3ce4-4aab-8164-5b8ced7e9175 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:43:57.628Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=239d356b-3ce4-4aab-8164-5b8ced7e9175 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:43:59.834Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b9b4c35c-4ff1-4f96-8130-070fb7b7d175 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:43:59.834Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b9b4c35c-4ff1-4f96-8130-070fb7b7d175 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3cba53ff 2021-07-13T00:43:59.834Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b9b4c35c-4ff1-4f96-8130-070fb7b7d175 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:43:59.834Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b9b4c35c-4ff1-4f96-8130-070fb7b7d175 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:00.423Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b9b4c35c-4ff1-4f96-8130-070fb7b7d175 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:44:00.423Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b9b4c35c-4ff1-4f96-8130-070fb7b7d175 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:44:01.826Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdd03420-d005-4602-8a6a-c13b09abd6ca PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:01.826Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdd03420-d005-4602-8a6a-c13b09abd6ca PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@509634b0 2021-07-13T00:44:01.826Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdd03420-d005-4602-8a6a-c13b09abd6ca PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:01.826Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cdd03420-d005-4602-8a6a-c13b09abd6ca PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:04.720Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f0307d2-3af9-4827-b663-b97cbf456620 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:04.720Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f0307d2-3af9-4827-b663-b97cbf456620 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@54081e00 2021-07-13T00:44:04.720Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f0307d2-3af9-4827-b663-b97cbf456620 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:04.720Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f0307d2-3af9-4827-b663-b97cbf456620 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:07.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:44:07.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:44:07.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:44:07.919Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28b01a90-b1b4-4c47-a52b-69482dab4c90 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:07.919Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28b01a90-b1b4-4c47-a52b-69482dab4c90 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@323681c6 2021-07-13T00:44:07.919Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28b01a90-b1b4-4c47-a52b-69482dab4c90 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:07.919Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=28b01a90-b1b4-4c47-a52b-69482dab4c90 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:07.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:44:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:44:09.369Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:44:09.369Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:44:09.559Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:44:09.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abfb9733-a55e-4fab-b71e-e4e88974fd17 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:09.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abfb9733-a55e-4fab-b71e-e4e88974fd17 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5cfd19d4 2021-07-13T00:44:09.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abfb9733-a55e-4fab-b71e-e4e88974fd17 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:09.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abfb9733-a55e-4fab-b71e-e4e88974fd17 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:10.318Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=abfb9733-a55e-4fab-b71e-e4e88974fd17 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:44:10.318Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=abfb9733-a55e-4fab-b71e-e4e88974fd17 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:44:11.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1933bcee-0e64-4c88-95ee-0d4b35e0f2ac PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:11.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1933bcee-0e64-4c88-95ee-0d4b35e0f2ac PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3daf954f 2021-07-13T00:44:11.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1933bcee-0e64-4c88-95ee-0d4b35e0f2ac PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:11.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1933bcee-0e64-4c88-95ee-0d4b35e0f2ac PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:12.562Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=1933bcee-0e64-4c88-95ee-0d4b35e0f2ac ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:44:12.563Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=1933bcee-0e64-4c88-95ee-0d4b35e0f2ac ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:44:13.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78e3dec9-c4c0-4a4d-a042-0ffc5fd37a93 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:13.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78e3dec9-c4c0-4a4d-a042-0ffc5fd37a93 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@29c374e4 2021-07-13T00:44:13.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78e3dec9-c4c0-4a4d-a042-0ffc5fd37a93 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:13.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78e3dec9-c4c0-4a4d-a042-0ffc5fd37a93 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:15.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0da2746a-971a-43f9-b7ae-fba44fa82e4e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:15.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0da2746a-971a-43f9-b7ae-fba44fa82e4e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ffde8ab 2021-07-13T00:44:15.372Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0da2746a-971a-43f9-b7ae-fba44fa82e4e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:15.372Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0da2746a-971a-43f9-b7ae-fba44fa82e4e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:17.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e36cc37-19c0-4e65-b261-709a59b27fcd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:17.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e36cc37-19c0-4e65-b261-709a59b27fcd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b7a7739 2021-07-13T00:44:17.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e36cc37-19c0-4e65-b261-709a59b27fcd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:17.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9e36cc37-19c0-4e65-b261-709a59b27fcd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:18.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=825d22cd-e752-4ca2-a84e-44b8ec3e6902 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:18.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=825d22cd-e752-4ca2-a84e-44b8ec3e6902 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5a3e9cf4 2021-07-13T00:44:18.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=825d22cd-e752-4ca2-a84e-44b8ec3e6902 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:18.569Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=825d22cd-e752-4ca2-a84e-44b8ec3e6902 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:20.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71e98d74-e962-4a25-aee7-a37fa24ddd2f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:20.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71e98d74-e962-4a25-aee7-a37fa24ddd2f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@68e8a703 2021-07-13T00:44:20.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71e98d74-e962-4a25-aee7-a37fa24ddd2f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:20.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=71e98d74-e962-4a25-aee7-a37fa24ddd2f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:22.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41ed401f-b4b4-49b3-9557-d32b4902d5bc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:22.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41ed401f-b4b4-49b3-9557-d32b4902d5bc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2810b205 2021-07-13T00:44:22.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41ed401f-b4b4-49b3-9557-d32b4902d5bc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:22.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41ed401f-b4b4-49b3-9557-d32b4902d5bc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:22.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:44:22.927Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:44:23.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f09fab50-5b9f-49c5-b061-e26b2b5a2db8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:23.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f09fab50-5b9f-49c5-b061-e26b2b5a2db8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@601bdb81 2021-07-13T00:44:23.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f09fab50-5b9f-49c5-b061-e26b2b5a2db8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:23.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f09fab50-5b9f-49c5-b061-e26b2b5a2db8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:24.775Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:44:25.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f87c8a5e-d30f-44d6-8790-7b309ca197a3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:25.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f87c8a5e-d30f-44d6-8790-7b309ca197a3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4e6c324a 2021-07-13T00:44:25.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f87c8a5e-d30f-44d6-8790-7b309ca197a3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:25.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f87c8a5e-d30f-44d6-8790-7b309ca197a3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:27.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d50c78f-28b1-41f6-984f-c94842a37976 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:27.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d50c78f-28b1-41f6-984f-c94842a37976 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6311443b 2021-07-13T00:44:27.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d50c78f-28b1-41f6-984f-c94842a37976 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:27.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d50c78f-28b1-41f6-984f-c94842a37976 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:29.174Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a30fff1f-90b5-4669-873b-92fb85dbf771 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:29.174Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a30fff1f-90b5-4669-873b-92fb85dbf771 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ebc52c6 2021-07-13T00:44:29.174Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a30fff1f-90b5-4669-873b-92fb85dbf771 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:29.174Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a30fff1f-90b5-4669-873b-92fb85dbf771 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:30.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=100864bb-3c92-4cbb-b647-2c0bb70f8251 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:30.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=100864bb-3c92-4cbb-b647-2c0bb70f8251 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@70c54f43 2021-07-13T00:44:30.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=100864bb-3c92-4cbb-b647-2c0bb70f8251 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:30.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=100864bb-3c92-4cbb-b647-2c0bb70f8251 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:32.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a0fff52d-1f6b-492c-a37b-a78160d2de20 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:32.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a0fff52d-1f6b-492c-a37b-a78160d2de20 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f94cd73 2021-07-13T00:44:32.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a0fff52d-1f6b-492c-a37b-a78160d2de20 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:32.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a0fff52d-1f6b-492c-a37b-a78160d2de20 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:34.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55dbb89a-becf-4325-8e81-68e31a3716d3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:34.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55dbb89a-becf-4325-8e81-68e31a3716d3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@364d6a26 2021-07-13T00:44:34.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55dbb89a-becf-4325-8e81-68e31a3716d3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:34.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=55dbb89a-becf-4325-8e81-68e31a3716d3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:35.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c827f41b-972a-4c5a-b8e3-4931abc2ec5e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:35.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c827f41b-972a-4c5a-b8e3-4931abc2ec5e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1acb344b 2021-07-13T00:44:35.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c827f41b-972a-4c5a-b8e3-4931abc2ec5e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:35.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c827f41b-972a-4c5a-b8e3-4931abc2ec5e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:44:37.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:44:38.013Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=77b6c910-e564-4e54-a694-4e0b86cf4c2b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:38.013Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=77b6c910-e564-4e54-a694-4e0b86cf4c2b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@40aedf06 2021-07-13T00:44:38.013Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=77b6c910-e564-4e54-a694-4e0b86cf4c2b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:38.013Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=77b6c910-e564-4e54-a694-4e0b86cf4c2b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:39.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9cf8997-45c7-4bae-9f6b-f0d393d4ab2c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:39.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9cf8997-45c7-4bae-9f6b-f0d393d4ab2c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@39a86d35 2021-07-13T00:44:39.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9cf8997-45c7-4bae-9f6b-f0d393d4ab2c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:39.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9cf8997-45c7-4bae-9f6b-f0d393d4ab2c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:41.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3be7786a-1625-4966-912f-b58773020153 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:41.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3be7786a-1625-4966-912f-b58773020153 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64692c7c 2021-07-13T00:44:41.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3be7786a-1625-4966-912f-b58773020153 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:41.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3be7786a-1625-4966-912f-b58773020153 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:43.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4412538-1896-4a1e-82a6-3c29fa468214 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:43.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4412538-1896-4a1e-82a6-3c29fa468214 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@274335d6 2021-07-13T00:44:43.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4412538-1896-4a1e-82a6-3c29fa468214 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:43.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4412538-1896-4a1e-82a6-3c29fa468214 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:45.362Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad5ba001-8981-4ca2-9411-faac84d4784e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:45.362Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad5ba001-8981-4ca2-9411-faac84d4784e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@11a398d7 2021-07-13T00:44:45.362Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad5ba001-8981-4ca2-9411-faac84d4784e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:45.362Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad5ba001-8981-4ca2-9411-faac84d4784e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:46.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e8fdb2d-82e6-451a-82a1-2011821f08bb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:46.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e8fdb2d-82e6-451a-82a1-2011821f08bb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6b189e13 2021-07-13T00:44:46.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e8fdb2d-82e6-451a-82a1-2011821f08bb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:46.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e8fdb2d-82e6-451a-82a1-2011821f08bb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:48.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1540c01a-1020-43b1-9e4a-718117cfc82a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:48.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1540c01a-1020-43b1-9e4a-718117cfc82a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@46bf55c0 2021-07-13T00:44:48.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1540c01a-1020-43b1-9e4a-718117cfc82a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:48.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1540c01a-1020-43b1-9e4a-718117cfc82a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:50.476Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f9e1e49-7e5f-47d0-a837-74aafbcf3d3d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:50.476Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f9e1e49-7e5f-47d0-a837-74aafbcf3d3d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78ec2544 2021-07-13T00:44:50.476Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f9e1e49-7e5f-47d0-a837-74aafbcf3d3d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:50.476Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f9e1e49-7e5f-47d0-a837-74aafbcf3d3d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:52.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c631f02-74d8-48c5-8abb-6a1bef797377 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:52.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c631f02-74d8-48c5-8abb-6a1bef797377 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@75846fae 2021-07-13T00:44:52.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c631f02-74d8-48c5-8abb-6a1bef797377 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:52.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c631f02-74d8-48c5-8abb-6a1bef797377 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:44:52.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:44:54.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40922282-83b1-4ebe-8ea3-a1450f5d2589 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:54.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40922282-83b1-4ebe-8ea3-a1450f5d2589 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5ac38f0b 2021-07-13T00:44:54.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40922282-83b1-4ebe-8ea3-a1450f5d2589 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:54.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40922282-83b1-4ebe-8ea3-a1450f5d2589 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:55.370Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81a495a5-628a-40d6-84c5-6c48ebb10374 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:55.370Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81a495a5-628a-40d6-84c5-6c48ebb10374 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@66eed0c6 2021-07-13T00:44:55.370Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81a495a5-628a-40d6-84c5-6c48ebb10374 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:55.370Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81a495a5-628a-40d6-84c5-6c48ebb10374 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:57.178Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=39e6381b-0238-48a3-807a-b3cb78849c8a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:57.178Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=39e6381b-0238-48a3-807a-b3cb78849c8a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f310f5a 2021-07-13T00:44:57.179Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=39e6381b-0238-48a3-807a-b3cb78849c8a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:57.179Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=39e6381b-0238-48a3-807a-b3cb78849c8a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:44:58.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0053511c-f5cb-47e7-8e4b-7bc92d75a9ea PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:44:58.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0053511c-f5cb-47e7-8e4b-7bc92d75a9ea PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@299ef60d 2021-07-13T00:44:58.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0053511c-f5cb-47e7-8e4b-7bc92d75a9ea PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:44:58.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0053511c-f5cb-47e7-8e4b-7bc92d75a9ea PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:00.926Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e50c90e-3403-4b55-b5da-925e1925bbbd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:00.926Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e50c90e-3403-4b55-b5da-925e1925bbbd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6fe38fe2 2021-07-13T00:45:00.926Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e50c90e-3403-4b55-b5da-925e1925bbbd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:00.926Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e50c90e-3403-4b55-b5da-925e1925bbbd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:03.428Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6660d026-2d87-49c6-97f6-53c4b36d27a5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:03.429Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6660d026-2d87-49c6-97f6-53c4b36d27a5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f1b0a38 2021-07-13T00:45:03.429Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6660d026-2d87-49c6-97f6-53c4b36d27a5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:03.429Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6660d026-2d87-49c6-97f6-53c4b36d27a5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:05.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91914d0f-b815-4cd9-b9aa-453bc9c46b7c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:05.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91914d0f-b815-4cd9-b9aa-453bc9c46b7c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@827815e 2021-07-13T00:45:05.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91914d0f-b815-4cd9-b9aa-453bc9c46b7c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:05.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91914d0f-b815-4cd9-b9aa-453bc9c46b7c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:07.426Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9edbf4c-4de2-45d9-9b29-f4498a6fafd0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:07.426Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9edbf4c-4de2-45d9-9b29-f4498a6fafd0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@565720c9 2021-07-13T00:45:07.426Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9edbf4c-4de2-45d9-9b29-f4498a6fafd0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:07.426Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d9edbf4c-4de2-45d9-9b29-f4498a6fafd0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:45:07.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:45:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:45:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:45:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:45:09.556Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=488a9dbc-97f8-4fe6-bee0-e3802804478d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:09.556Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=488a9dbc-97f8-4fe6-bee0-e3802804478d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@25b8397e 2021-07-13T00:45:09.556Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=488a9dbc-97f8-4fe6-bee0-e3802804478d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:09.556Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=488a9dbc-97f8-4fe6-bee0-e3802804478d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:11.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f044864-a20e-48b8-9c63-b0f92800f9f8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:11.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f044864-a20e-48b8-9c63-b0f92800f9f8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d5ad3b 2021-07-13T00:45:11.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f044864-a20e-48b8-9c63-b0f92800f9f8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:11.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f044864-a20e-48b8-9c63-b0f92800f9f8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:13.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d678c226-d603-4c95-a757-0150b74df9e6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:13.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d678c226-d603-4c95-a757-0150b74df9e6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@364ce14e 2021-07-13T00:45:13.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d678c226-d603-4c95-a757-0150b74df9e6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:13.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d678c226-d603-4c95-a757-0150b74df9e6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:15.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4d1e3b5-2ce1-4abb-bece-ca710ada8749 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:15.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4d1e3b5-2ce1-4abb-bece-ca710ada8749 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5972de26 2021-07-13T00:45:15.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4d1e3b5-2ce1-4abb-bece-ca710ada8749 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:15.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4d1e3b5-2ce1-4abb-bece-ca710ada8749 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:17.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4fafa954-ca34-4d20-83a6-967a363fd030 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:17.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4fafa954-ca34-4d20-83a6-967a363fd030 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3d656fbf 2021-07-13T00:45:17.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4fafa954-ca34-4d20-83a6-967a363fd030 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:17.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4fafa954-ca34-4d20-83a6-967a363fd030 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:20.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3c46ffc-4a21-4d2f-ac96-f9a5d69f5905 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:20.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3c46ffc-4a21-4d2f-ac96-f9a5d69f5905 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@12f6a15d 2021-07-13T00:45:20.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3c46ffc-4a21-4d2f-ac96-f9a5d69f5905 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:20.371Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3c46ffc-4a21-4d2f-ac96-f9a5d69f5905 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:22.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:45:22.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:45:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:45:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:45:23.584Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5055e77d-75fe-462d-8c86-3856f1805ba8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:23.584Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5055e77d-75fe-462d-8c86-3856f1805ba8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@43e174dc 2021-07-13T00:45:23.584Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5055e77d-75fe-462d-8c86-3856f1805ba8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:23.584Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5055e77d-75fe-462d-8c86-3856f1805ba8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:24.474Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=5055e77d-75fe-462d-8c86-3856f1805ba8 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:45:24.474Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=5055e77d-75fe-462d-8c86-3856f1805ba8 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:45:24.957Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:45:25.206Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:45:25.656Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=276e1e72-1226-4208-befa-5e6bcaf500c9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:25.656Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=276e1e72-1226-4208-befa-5e6bcaf500c9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@cad1168 2021-07-13T00:45:25.656Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=276e1e72-1226-4208-befa-5e6bcaf500c9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:25.657Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=276e1e72-1226-4208-befa-5e6bcaf500c9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:27.363Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0005fd30-369b-4a59-941b-df5463b626fd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:27.363Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0005fd30-369b-4a59-941b-df5463b626fd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1aa38fd5 2021-07-13T00:45:27.363Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0005fd30-369b-4a59-941b-df5463b626fd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:27.363Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0005fd30-369b-4a59-941b-df5463b626fd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:29.360Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=0005fd30-369b-4a59-941b-df5463b626fd ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:45:29.360Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=0005fd30-369b-4a59-941b-df5463b626fd ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:45:30.420Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=23a59059-f8cf-4ba5-ac00-555cda8f8cb3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:30.420Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=23a59059-f8cf-4ba5-ac00-555cda8f8cb3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3df82995 2021-07-13T00:45:30.420Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=23a59059-f8cf-4ba5-ac00-555cda8f8cb3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:30.420Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=23a59059-f8cf-4ba5-ac00-555cda8f8cb3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:32.932Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d45fbd92-8ce9-4d32-8879-856368b0c366 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:32.932Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d45fbd92-8ce9-4d32-8879-856368b0c366 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@75f7fd62 2021-07-13T00:45:32.932Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d45fbd92-8ce9-4d32-8879-856368b0c366 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:32.932Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d45fbd92-8ce9-4d32-8879-856368b0c366 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:36.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41251869-4d78-4be8-bdf4-5ba7ecd29257 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:36.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41251869-4d78-4be8-bdf4-5ba7ecd29257 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@113584e5 2021-07-13T00:45:36.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41251869-4d78-4be8-bdf4-5ba7ecd29257 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:36.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41251869-4d78-4be8-bdf4-5ba7ecd29257 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:37.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:45:37.918Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:45:39.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=758bb6cd-6309-48f5-b612-b792f01246c3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:39.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=758bb6cd-6309-48f5-b612-b792f01246c3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@176e6c31 2021-07-13T00:45:39.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=758bb6cd-6309-48f5-b612-b792f01246c3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:39.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=758bb6cd-6309-48f5-b612-b792f01246c3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:41.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a38175a5-920e-4f94-a0dc-0854cb9e8a58 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:41.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a38175a5-920e-4f94-a0dc-0854cb9e8a58 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6b265bc8 2021-07-13T00:45:41.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a38175a5-920e-4f94-a0dc-0854cb9e8a58 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:41.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a38175a5-920e-4f94-a0dc-0854cb9e8a58 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:44.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c86bd090-ea63-44e6-a231-75cf7dc5f152 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:44.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c86bd090-ea63-44e6-a231-75cf7dc5f152 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@21f0b723 2021-07-13T00:45:44.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c86bd090-ea63-44e6-a231-75cf7dc5f152 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:44.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c86bd090-ea63-44e6-a231-75cf7dc5f152 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:48.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8add5105-f142-4858-992b-52ff6e1a2fce PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:48.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8add5105-f142-4858-992b-52ff6e1a2fce PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1cdf0d54 2021-07-13T00:45:48.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8add5105-f142-4858-992b-52ff6e1a2fce PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:48.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8add5105-f142-4858-992b-52ff6e1a2fce PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:50.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ab7b8f1-464e-4869-910a-53816e2e13ae PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:50.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ab7b8f1-464e-4869-910a-53816e2e13ae PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78b4e51a 2021-07-13T00:45:50.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ab7b8f1-464e-4869-910a-53816e2e13ae PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:50.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ab7b8f1-464e-4869-910a-53816e2e13ae PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:52.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:45:52.926Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:45:53.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb5dd520-4b35-4fda-aae6-e126e7698c8c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:53.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb5dd520-4b35-4fda-aae6-e126e7698c8c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@219509df 2021-07-13T00:45:53.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb5dd520-4b35-4fda-aae6-e126e7698c8c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:53.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb5dd520-4b35-4fda-aae6-e126e7698c8c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:56.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f2836f1b-2dc2-44bf-93e7-fcbee195b990 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:56.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f2836f1b-2dc2-44bf-93e7-fcbee195b990 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@578fb61 2021-07-13T00:45:56.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f2836f1b-2dc2-44bf-93e7-fcbee195b990 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:56.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f2836f1b-2dc2-44bf-93e7-fcbee195b990 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:45:59.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c06fbb3-dde5-4fd1-bcdb-f6bfa500dad1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:45:59.969Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c06fbb3-dde5-4fd1-bcdb-f6bfa500dad1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6129e337 2021-07-13T00:45:59.969Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c06fbb3-dde5-4fd1-bcdb-f6bfa500dad1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:45:59.969Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c06fbb3-dde5-4fd1-bcdb-f6bfa500dad1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:46:02.575Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6bab5ce-60c5-4b8b-a005-f15e19e16024 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:46:02.575Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6bab5ce-60c5-4b8b-a005-f15e19e16024 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3e6a563d 2021-07-13T00:46:02.575Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6bab5ce-60c5-4b8b-a005-f15e19e16024 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:46:02.575Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6bab5ce-60c5-4b8b-a005-f15e19e16024 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:46:05.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38bfd763-98cf-4415-ada1-daf4729b295b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:46:05.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38bfd763-98cf-4415-ada1-daf4729b295b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@37b87bde 2021-07-13T00:46:05.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38bfd763-98cf-4415-ada1-daf4729b295b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:46:05.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=38bfd763-98cf-4415-ada1-daf4729b295b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:46:07.917Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:46:07.925Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:46:08.770Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03ed22cd-a20b-4daa-a692-8d09faeeca83 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:46:08.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03ed22cd-a20b-4daa-a692-8d09faeeca83 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@75d92dbb 2021-07-13T00:46:08.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03ed22cd-a20b-4daa-a692-8d09faeeca83 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:46:08.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03ed22cd-a20b-4daa-a692-8d09faeeca83 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:46:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:46:09.369Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:46:09.369Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:46:11.926Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4d9317b-80ec-4f20-89c3-d964828ac432 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:46:11.926Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4d9317b-80ec-4f20-89c3-d964828ac432 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5b47586c 2021-07-13T00:46:11.926Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4d9317b-80ec-4f20-89c3-d964828ac432 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:46:11.926Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e4d9317b-80ec-4f20-89c3-d964828ac432 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:46:13.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:46:13.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:46:14.932Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=e4d9317b-80ec-4f20-89c3-d964828ac432 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:46:14.934Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=e4d9317b-80ec-4f20-89c3-d964828ac432 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:46:14.934Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=e4d9317b-80ec-4f20-89c3-d964828ac432 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:46:14.934Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=e4d9317b-80ec-4f20-89c3-d964828ac432 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:46:16.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:46:17.323Z [pool-8-thread-1] WARN o.j.diskstorage.log.kcvs.KCVSLog Could not read messages for timestamp [2021-07-13T00:46:06.467337Z] (this read will be retried) org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:56) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:726) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:100) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) ... 8 common frames omitted Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:792) at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:789) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) ... 9 common frames omitted Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 24 common frames omitted 2021-07-13T00:46:17.323Z [ApplicationDataTypeCacheThread-1] INFO o.j.d.util.BackendOperation RequestID=null ErrorCategory=INFO ServiceName=ReplaceDataTypesCache TargetServiceName=null ErrorCode=900 TargetEntity=ApplicationDataTypesCache Temporary exception during backend operation [EdgeStoreQuery]. Attempting backoff retry. org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174) at io.vavr.API$Match.of(API.java:3137) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:123) at io.vavr.control.Try.getOrElseThrow(Try.java:671) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:77) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:272) at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:269) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:470) at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:269) at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:436) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:98) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:90) at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:82) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:67) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:120) at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:77) at org.janusgraph.graphdb.transaction.RelationConstructor$1.(RelationConstructor.java:47) at org.janusgraph.graphdb.transaction.RelationConstructor.lambda$readRelation$0(RelationConstructor.java:45) at org.janusgraph.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:177) at org.apache.tinkerpop.gremlin.structure.Element.keys(Element.java:69) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getProperties(JanusGraphGenericDao.java:1491) at org.openecomp.sdc.be.dao.janusgraph.JanusGraphGenericDao.getByCriteria(JanusGraphGenericDao.java:759) at org.openecomp.sdc.be.model.operations.impl.PropertyOperation.getAllDataTypeNodes(PropertyOperation.java:1566) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.findAllDataTypesLazy(ApplicationDataTypeCache.java:309) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.hasDataTypesChanged(ApplicationDataTypeCache.java:220) at org.openecomp.sdc.be.model.cache.ApplicationDataTypeCache.run(ApplicationDataTypeCache.java:200) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547) at io.vavr.control.Try.of(Try.java:62) at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124) at com.datastax.driver.core.Responses$Error.asException(Responses.java:169) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:645) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:308) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:422) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 common frames omitted Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:90) at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:65) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:297) at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:268) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 26 common frames omitted 2021-07-13T00:46:18.115Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e93e622b-751e-4c79-a1ab-190d19533b4b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:46:18.115Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e93e622b-751e-4c79-a1ab-190d19533b4b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@37f03687 2021-07-13T00:46:18.115Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e93e622b-751e-4c79-a1ab-190d19533b4b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:46:18.115Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e93e622b-751e-4c79-a1ab-190d19533b4b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:46:18.586Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=e93e622b-751e-4c79-a1ab-190d19533b4b ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:46:18.586Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=e93e622b-751e-4c79-a1ab-190d19533b4b ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:46:18.668Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:46:19.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92d599a0-7f0a-46eb-af12-5e64d93ddadb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:46:19.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92d599a0-7f0a-46eb-af12-5e64d93ddadb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7fea840e 2021-07-13T00:46:19.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92d599a0-7f0a-46eb-af12-5e64d93ddadb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:46:19.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92d599a0-7f0a-46eb-af12-5e64d93ddadb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:46:21.524Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e64a6bc7-0bcf-4f7e-be21-3b0abbd17b40 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:46:21.524Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e64a6bc7-0bcf-4f7e-be21-3b0abbd17b40 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1e0f27a7 2021-07-13T00:46:21.524Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e64a6bc7-0bcf-4f7e-be21-3b0abbd17b40 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:46:21.524Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e64a6bc7-0bcf-4f7e-be21-3b0abbd17b40 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:46:21.866Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=e64a6bc7-0bcf-4f7e-be21-3b0abbd17b40 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:46:21.866Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=e64a6bc7-0bcf-4f7e-be21-3b0abbd17b40 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:46:22.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:46:22.932Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:46:23.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=859d3967-8e70-47cf-8177-593ea32a2ba8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:46:23.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=859d3967-8e70-47cf-8177-593ea32a2ba8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@28fd7eef 2021-07-13T00:46:23.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=859d3967-8e70-47cf-8177-593ea32a2ba8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:46:23.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=859d3967-8e70-47cf-8177-593ea32a2ba8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:46:25.043Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:46:26.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=853f554e-c1e0-4261-9ca1-a634cc9a9e5e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:46:26.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=853f554e-c1e0-4261-9ca1-a634cc9a9e5e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7da34d61 2021-07-13T00:46:26.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=853f554e-c1e0-4261-9ca1-a634cc9a9e5e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:46:26.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=853f554e-c1e0-4261-9ca1-a634cc9a9e5e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:46:29.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5c43e95-95ab-435a-b1af-12f4e82082d4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:46:29.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5c43e95-95ab-435a-b1af-12f4e82082d4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@56c26c2f 2021-07-13T00:46:29.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5c43e95-95ab-435a-b1af-12f4e82082d4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:46:29.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d5c43e95-95ab-435a-b1af-12f4e82082d4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:46:32.910Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d96e00ea-cecd-4843-845a-852e4a9ee96e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:46:32.910Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d96e00ea-cecd-4843-845a-852e4a9ee96e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@31174b4f 2021-07-13T00:46:32.910Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d96e00ea-cecd-4843-845a-852e4a9ee96e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:46:32.910Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d96e00ea-cecd-4843-845a-852e4a9ee96e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:46:35.670Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=45fc2464-f2d2-4a0a-ab3a-619783c89ce1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:46:35.670Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=45fc2464-f2d2-4a0a-ab3a-619783c89ce1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@72de5184 2021-07-13T00:46:35.670Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=45fc2464-f2d2-4a0a-ab3a-619783c89ce1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:46:35.670Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=45fc2464-f2d2-4a0a-ab3a-619783c89ce1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:46:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:46:37.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:46:39.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=078d606b-fd68-4b43-a8dc-242987bf26ff PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:46:39.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=078d606b-fd68-4b43-a8dc-242987bf26ff PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@15f1d2fb 2021-07-13T00:46:39.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=078d606b-fd68-4b43-a8dc-242987bf26ff PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:46:39.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=078d606b-fd68-4b43-a8dc-242987bf26ff PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:46:41.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=520a1843-777f-4b2e-a3dd-3af8ccc22ac1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:46:41.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=520a1843-777f-4b2e-a3dd-3af8ccc22ac1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5017777e 2021-07-13T00:46:41.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=520a1843-777f-4b2e-a3dd-3af8ccc22ac1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:46:41.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=520a1843-777f-4b2e-a3dd-3af8ccc22ac1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:46:44.879Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3f361a8-9bba-48f6-87c3-e6d4cf7ff38c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:46:44.879Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3f361a8-9bba-48f6-87c3-e6d4cf7ff38c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5b511300 2021-07-13T00:46:44.879Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3f361a8-9bba-48f6-87c3-e6d4cf7ff38c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:46:44.879Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c3f361a8-9bba-48f6-87c3-e6d4cf7ff38c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:46:47.884Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=c3f361a8-9bba-48f6-87c3-e6d4cf7ff38c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:46:47.884Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=c3f361a8-9bba-48f6-87c3-e6d4cf7ff38c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:46:47.884Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c3f361a8-9bba-48f6-87c3-e6d4cf7ff38c ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:46:47.884Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c3f361a8-9bba-48f6-87c3-e6d4cf7ff38c ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:46:49.728Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13294a34-c152-4266-b3f0-7f22d6983d4b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:46:49.728Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13294a34-c152-4266-b3f0-7f22d6983d4b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5b832b9f 2021-07-13T00:46:49.728Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13294a34-c152-4266-b3f0-7f22d6983d4b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:46:49.728Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13294a34-c152-4266-b3f0-7f22d6983d4b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:46:50.266Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=13294a34-c152-4266-b3f0-7f22d6983d4b ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:46:50.266Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=13294a34-c152-4266-b3f0-7f22d6983d4b ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:46:51.373Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d7d531d-fe41-4078-84e3-6ac0d617909f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:46:51.373Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d7d531d-fe41-4078-84e3-6ac0d617909f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2b396ff8 2021-07-13T00:46:51.373Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d7d531d-fe41-4078-84e3-6ac0d617909f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:46:51.373Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d7d531d-fe41-4078-84e3-6ac0d617909f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:46:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:46:52.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:46:54.617Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8cfc2e4d-0011-41e4-a0b4-32cd34803483 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:46:54.617Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8cfc2e4d-0011-41e4-a0b4-32cd34803483 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78a3b08c 2021-07-13T00:46:54.617Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8cfc2e4d-0011-41e4-a0b4-32cd34803483 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:46:54.617Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8cfc2e4d-0011-41e4-a0b4-32cd34803483 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:46:56.814Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08baa6dd-8a61-4cf1-8d66-797375879151 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:46:56.814Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08baa6dd-8a61-4cf1-8d66-797375879151 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3a3eb3d4 2021-07-13T00:46:56.814Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08baa6dd-8a61-4cf1-8d66-797375879151 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:46:56.814Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08baa6dd-8a61-4cf1-8d66-797375879151 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:47:00.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97b70ba0-464d-4c72-8d10-f3210bf49ded PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:47:00.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97b70ba0-464d-4c72-8d10-f3210bf49ded PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@8129631 2021-07-13T00:47:00.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97b70ba0-464d-4c72-8d10-f3210bf49ded PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:47:00.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=97b70ba0-464d-4c72-8d10-f3210bf49ded PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:47:03.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50a3964c-bb1f-4455-8a44-8748aeccac16 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:47:03.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50a3964c-bb1f-4455-8a44-8748aeccac16 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@68b46e64 2021-07-13T00:47:03.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50a3964c-bb1f-4455-8a44-8748aeccac16 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:47:03.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=50a3964c-bb1f-4455-8a44-8748aeccac16 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:47:06.325Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1915f5b4-296f-4fba-96a9-6648b30b8ff1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:47:06.325Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1915f5b4-296f-4fba-96a9-6648b30b8ff1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@df6942e 2021-07-13T00:47:06.325Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1915f5b4-296f-4fba-96a9-6648b30b8ff1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:47:06.325Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1915f5b4-296f-4fba-96a9-6648b30b8ff1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:47:07.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:47:07.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:47:08.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15c2b6fb-3571-44e2-9a69-10257bf67dae PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:47:08.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15c2b6fb-3571-44e2-9a69-10257bf67dae PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@42992657 2021-07-13T00:47:08.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15c2b6fb-3571-44e2-9a69-10257bf67dae PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:47:08.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=15c2b6fb-3571-44e2-9a69-10257bf67dae PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:47:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:47:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:47:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:47:12.157Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4941df84-ef37-4958-9794-2a65356c4465 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:47:12.157Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4941df84-ef37-4958-9794-2a65356c4465 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5e26832b 2021-07-13T00:47:12.157Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4941df84-ef37-4958-9794-2a65356c4465 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:47:12.157Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4941df84-ef37-4958-9794-2a65356c4465 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:47:15.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70178356-99a8-4667-ad3f-583fcf97944c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:47:15.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70178356-99a8-4667-ad3f-583fcf97944c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@152a11c5 2021-07-13T00:47:15.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70178356-99a8-4667-ad3f-583fcf97944c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:47:15.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70178356-99a8-4667-ad3f-583fcf97944c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:47:18.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9cda52e5-a6fc-49f6-aa0c-8ece631cc562 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:47:18.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9cda52e5-a6fc-49f6-aa0c-8ece631cc562 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2cc4a0aa 2021-07-13T00:47:18.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9cda52e5-a6fc-49f6-aa0c-8ece631cc562 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:47:18.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9cda52e5-a6fc-49f6-aa0c-8ece631cc562 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:47:19.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:47:19.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:47:21.164Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=9cda52e5-a6fc-49f6-aa0c-8ece631cc562 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:47:21.164Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=9cda52e5-a6fc-49f6-aa0c-8ece631cc562 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:47:21.164Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9cda52e5-a6fc-49f6-aa0c-8ece631cc562 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:47:21.164Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9cda52e5-a6fc-49f6-aa0c-8ece631cc562 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:47:22.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:47:22.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:47:22.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:47:24.664Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:47:24.814Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d488dea-f3ac-4375-a0b0-dbcc33c87e68 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:47:24.814Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d488dea-f3ac-4375-a0b0-dbcc33c87e68 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@276480fb 2021-07-13T00:47:24.814Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d488dea-f3ac-4375-a0b0-dbcc33c87e68 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:47:24.814Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6d488dea-f3ac-4375-a0b0-dbcc33c87e68 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:47:25.520Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6d488dea-f3ac-4375-a0b0-dbcc33c87e68 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:47:25.520Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=6d488dea-f3ac-4375-a0b0-dbcc33c87e68 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:47:26.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8362b84-4601-4668-ba61-0b5128f5a199 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:47:26.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8362b84-4601-4668-ba61-0b5128f5a199 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@850e4ad 2021-07-13T00:47:26.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8362b84-4601-4668-ba61-0b5128f5a199 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:47:26.265Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8362b84-4601-4668-ba61-0b5128f5a199 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:47:27.064Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c8362b84-4601-4668-ba61-0b5128f5a199 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:47:27.064Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=c8362b84-4601-4668-ba61-0b5128f5a199 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:47:28.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc21e941-9669-4530-84c3-70bef538fa05 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:47:28.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc21e941-9669-4530-84c3-70bef538fa05 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@118d4fd 2021-07-13T00:47:28.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc21e941-9669-4530-84c3-70bef538fa05 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:47:28.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc21e941-9669-4530-84c3-70bef538fa05 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:47:29.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=347be82c-873d-47b6-83d6-09e0bdd90fce PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:47:29.849Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=347be82c-873d-47b6-83d6-09e0bdd90fce PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2f8ff7d 2021-07-13T00:47:29.850Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=347be82c-873d-47b6-83d6-09e0bdd90fce PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:47:29.850Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=347be82c-873d-47b6-83d6-09e0bdd90fce PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:47:33.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5776d51-bb6f-42a6-84e1-df3dd7266e5f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:47:33.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5776d51-bb6f-42a6-84e1-df3dd7266e5f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4408818d 2021-07-13T00:47:33.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5776d51-bb6f-42a6-84e1-df3dd7266e5f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:47:33.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5776d51-bb6f-42a6-84e1-df3dd7266e5f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:47:34.763Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:47:35.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3303e23-33c8-4656-a6f5-31a7b131f2cf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:47:35.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3303e23-33c8-4656-a6f5-31a7b131f2cf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4bf2602d 2021-07-13T00:47:35.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3303e23-33c8-4656-a6f5-31a7b131f2cf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:47:35.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3303e23-33c8-4656-a6f5-31a7b131f2cf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:47:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:47:37.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:47:39.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54283f77-a9d2-4ce0-bde6-51a93bddf4f1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:47:39.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54283f77-a9d2-4ce0-bde6-51a93bddf4f1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@c0aea92 2021-07-13T00:47:39.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54283f77-a9d2-4ce0-bde6-51a93bddf4f1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:47:39.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=54283f77-a9d2-4ce0-bde6-51a93bddf4f1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:47:41.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fbbffc2-11c4-4db4-bb2d-1094626be332 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:47:41.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fbbffc2-11c4-4db4-bb2d-1094626be332 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@49ccff79 2021-07-13T00:47:41.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fbbffc2-11c4-4db4-bb2d-1094626be332 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:47:41.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fbbffc2-11c4-4db4-bb2d-1094626be332 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:47:44.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc7eda70-ad1a-499f-83d9-2f97cf6b8435 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:47:44.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc7eda70-ad1a-499f-83d9-2f97cf6b8435 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@49480b24 2021-07-13T00:47:44.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc7eda70-ad1a-499f-83d9-2f97cf6b8435 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:47:44.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc7eda70-ad1a-499f-83d9-2f97cf6b8435 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:47:47.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=918ca5e5-9e53-49ad-853b-983a5d2a54be PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:47:47.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=918ca5e5-9e53-49ad-853b-983a5d2a54be PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1163f928 2021-07-13T00:47:47.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=918ca5e5-9e53-49ad-853b-983a5d2a54be PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:47:47.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=918ca5e5-9e53-49ad-853b-983a5d2a54be PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:47:51.317Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd419b1c-e54e-435a-a145-eca9449ab273 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:47:51.317Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd419b1c-e54e-435a-a145-eca9449ab273 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4a66cfec 2021-07-13T00:47:51.317Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd419b1c-e54e-435a-a145-eca9449ab273 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:47:51.317Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cd419b1c-e54e-435a-a145-eca9449ab273 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:47:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:47:52.926Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:47:53.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ae8b92f-f463-410d-8c33-55950a3f6f52 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:47:53.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ae8b92f-f463-410d-8c33-55950a3f6f52 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64ffd6ee 2021-07-13T00:47:53.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ae8b92f-f463-410d-8c33-55950a3f6f52 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:47:53.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6ae8b92f-f463-410d-8c33-55950a3f6f52 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:47:56.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f4e9361-79c8-431f-9852-b21d132992a4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:47:56.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f4e9361-79c8-431f-9852-b21d132992a4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6de7e07e 2021-07-13T00:47:56.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f4e9361-79c8-431f-9852-b21d132992a4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:47:56.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f4e9361-79c8-431f-9852-b21d132992a4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:48:00.126Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0faa1f2-ceb7-459f-a0eb-163bba9dc839 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:48:00.126Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0faa1f2-ceb7-459f-a0eb-163bba9dc839 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63c3fe53 2021-07-13T00:48:00.126Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0faa1f2-ceb7-459f-a0eb-163bba9dc839 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:48:00.126Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0faa1f2-ceb7-459f-a0eb-163bba9dc839 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:48:02.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=327a9597-7903-4d9b-9a6e-2e3fd9b93a8a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:48:02.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=327a9597-7903-4d9b-9a6e-2e3fd9b93a8a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f265468 2021-07-13T00:48:02.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=327a9597-7903-4d9b-9a6e-2e3fd9b93a8a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:48:02.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=327a9597-7903-4d9b-9a6e-2e3fd9b93a8a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:48:05.767Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=327a9597-7903-4d9b-9a6e-2e3fd9b93a8a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:48:05.768Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=327a9597-7903-4d9b-9a6e-2e3fd9b93a8a ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:48:05.768Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=327a9597-7903-4d9b-9a6e-2e3fd9b93a8a ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:48:05.768Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=327a9597-7903-4d9b-9a6e-2e3fd9b93a8a ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:48:06.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7564197b-9ff2-481d-b7f5-a08b365cae86 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:48:06.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7564197b-9ff2-481d-b7f5-a08b365cae86 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f4b9c4 2021-07-13T00:48:06.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7564197b-9ff2-481d-b7f5-a08b365cae86 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:48:06.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7564197b-9ff2-481d-b7f5-a08b365cae86 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:48:07.765Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=7564197b-9ff2-481d-b7f5-a08b365cae86 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:48:07.765Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=7564197b-9ff2-481d-b7f5-a08b365cae86 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:48:07.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:48:07.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:48:09.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2bdf3c19-f2de-4c1c-84a4-e7313731bac8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:48:09.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2bdf3c19-f2de-4c1c-84a4-e7313731bac8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2b8f10c9 2021-07-13T00:48:09.066Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2bdf3c19-f2de-4c1c-84a4-e7313731bac8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:48:09.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2bdf3c19-f2de-4c1c-84a4-e7313731bac8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:48:09.368Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:48:09.369Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:48:09.369Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:48:11.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b957f6f-acbb-4226-aa26-709ddb1a1212 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:48:11.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b957f6f-acbb-4226-aa26-709ddb1a1212 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@62784c4 2021-07-13T00:48:11.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b957f6f-acbb-4226-aa26-709ddb1a1212 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:48:11.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3b957f6f-acbb-4226-aa26-709ddb1a1212 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:48:14.775Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d32d118-d21c-456d-9af0-4acad79d5f46 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:48:14.775Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d32d118-d21c-456d-9af0-4acad79d5f46 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@263d275d 2021-07-13T00:48:14.775Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d32d118-d21c-456d-9af0-4acad79d5f46 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:48:14.776Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d32d118-d21c-456d-9af0-4acad79d5f46 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:48:19.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:48:19.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:48:22.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:48:22.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:48:22.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:48:25.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:48:27.517Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f840a457-6d7b-4172-8f70-9da92da095ac PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:48:27.517Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f840a457-6d7b-4172-8f70-9da92da095ac PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63527473 2021-07-13T00:48:27.517Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f840a457-6d7b-4172-8f70-9da92da095ac PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:48:27.517Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f840a457-6d7b-4172-8f70-9da92da095ac PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:48:27.663Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:48:28.311Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0130u5hm2m8hy31m5csqcec7fpe316 2021-07-13T00:48:28.311Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01fkix3qpswxsr1pxlrpq9m52o4307 2021-07-13T00:48:28.311Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0pf1pupw6xing1o1oty65ru7v322 2021-07-13T00:48:28.312Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0f3whb0hspaku1fcwzadylvj3m318 2021-07-13T00:48:28.312Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01tf64ioy9lkcclwghazx6n1f1312 2021-07-13T00:48:28.312Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0dltrar9lgibn1h50gfe6qkt1w320 2021-07-13T00:48:28.312Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01jg816ndtb2qhbo6muthwsnin315 2021-07-13T00:48:28.312Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node04wtt9kptrcj91955f52978urn319 2021-07-13T00:48:28.312Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01i9mfp118aiaj1se44s7chd9ds321 2021-07-13T00:48:28.312Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0qy0euq312gh2kmn7z7w40rzo308 2021-07-13T00:48:28.312Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node011id5ps2p7v2o1gwdtc01kcy8d310 2021-07-13T00:48:28.312Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node010hcb22gmbx4x1wczkzenny40b311 2021-07-13T00:48:28.312Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01ldhc3ffo8wj1zleyf5hxa6u0314 2021-07-13T00:48:28.312Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node017ypjoydc3n4c52r0o4nj8ti4313 2021-07-13T00:48:28.312Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node08yxisrsh5baprnjo72bj6zj8303 2021-07-13T00:48:28.312Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node05pabljll966k1k5q2mhgj6e7309 2021-07-13T00:48:28.312Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0qnzdn3vxstqz1gawmb4qsxuos306 2021-07-13T00:48:29.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3a8738c-fcb4-4953-b74c-c559cfe4e129 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:48:29.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3a8738c-fcb4-4953-b74c-c559cfe4e129 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@30083988 2021-07-13T00:48:29.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3a8738c-fcb4-4953-b74c-c559cfe4e129 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:48:29.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3a8738c-fcb4-4953-b74c-c559cfe4e129 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:48:29.711Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:48:33.374Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9e403f9-c57a-4b5d-84dd-c74df790b131 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:48:33.374Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9e403f9-c57a-4b5d-84dd-c74df790b131 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@a338ae9 2021-07-13T00:48:33.374Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9e403f9-c57a-4b5d-84dd-c74df790b131 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:48:33.374Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f9e403f9-c57a-4b5d-84dd-c74df790b131 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:48:37.468Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35768839-9c38-4231-86b3-7b429ce0dcd1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:48:37.468Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35768839-9c38-4231-86b3-7b429ce0dcd1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@45fb533d 2021-07-13T00:48:37.468Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35768839-9c38-4231-86b3-7b429ce0dcd1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:48:37.468Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35768839-9c38-4231-86b3-7b429ce0dcd1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:48:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:48:37.926Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:48:39.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d60c61cf-7a13-48e1-9346-ea5bc8cc6eab PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:48:39.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d60c61cf-7a13-48e1-9346-ea5bc8cc6eab PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@357a461d 2021-07-13T00:48:39.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d60c61cf-7a13-48e1-9346-ea5bc8cc6eab PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:48:39.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d60c61cf-7a13-48e1-9346-ea5bc8cc6eab PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:48:40.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be53b3ed-43d1-42c4-a379-5c3cf2296940 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:48:40.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be53b3ed-43d1-42c4-a379-5c3cf2296940 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@444ffd43 2021-07-13T00:48:40.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be53b3ed-43d1-42c4-a379-5c3cf2296940 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:48:40.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be53b3ed-43d1-42c4-a379-5c3cf2296940 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:48:42.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4e0d003-642e-49b1-82d4-9ce500f117e8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:48:42.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4e0d003-642e-49b1-82d4-9ce500f117e8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@523d43e1 2021-07-13T00:48:42.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4e0d003-642e-49b1-82d4-9ce500f117e8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:48:42.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f4e0d003-642e-49b1-82d4-9ce500f117e8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:48:43.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64c0cd4a-3a01-4c10-b735-42d846ea905b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:48:43.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64c0cd4a-3a01-4c10-b735-42d846ea905b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4dc08020 2021-07-13T00:48:43.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64c0cd4a-3a01-4c10-b735-42d846ea905b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:48:43.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64c0cd4a-3a01-4c10-b735-42d846ea905b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:48:45.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d84503b6-1720-44a5-8383-eaf046c911d9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:48:45.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d84503b6-1720-44a5-8383-eaf046c911d9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2a6d6d2b 2021-07-13T00:48:45.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d84503b6-1720-44a5-8383-eaf046c911d9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:48:45.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d84503b6-1720-44a5-8383-eaf046c911d9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:48:46.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=302cd9b9-62dc-42ee-9118-374588e08038 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:48:46.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=302cd9b9-62dc-42ee-9118-374588e08038 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@117338c0 2021-07-13T00:48:46.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=302cd9b9-62dc-42ee-9118-374588e08038 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:48:46.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=302cd9b9-62dc-42ee-9118-374588e08038 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:48:48.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c15bdb91-7c51-4ae9-9867-9aad2a088e7c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:48:48.360Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c15bdb91-7c51-4ae9-9867-9aad2a088e7c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@29695344 2021-07-13T00:48:48.360Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c15bdb91-7c51-4ae9-9867-9aad2a088e7c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:48:48.360Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c15bdb91-7c51-4ae9-9867-9aad2a088e7c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:48:50.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bbe75d2c-41e2-472e-a905-aff5ccf92bbb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:48:50.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bbe75d2c-41e2-472e-a905-aff5ccf92bbb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@60e117c9 2021-07-13T00:48:50.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bbe75d2c-41e2-472e-a905-aff5ccf92bbb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:48:50.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bbe75d2c-41e2-472e-a905-aff5ccf92bbb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:48:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:48:52.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:48:53.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1cecc7e4-5e11-4115-b7bf-07772f15e3bb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:48:53.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1cecc7e4-5e11-4115-b7bf-07772f15e3bb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@21242acb 2021-07-13T00:48:53.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1cecc7e4-5e11-4115-b7bf-07772f15e3bb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:48:53.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1cecc7e4-5e11-4115-b7bf-07772f15e3bb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:48:56.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25fa3975-c219-4b36-8260-4e9df55b1bd9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:48:56.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25fa3975-c219-4b36-8260-4e9df55b1bd9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@22be3fc4 2021-07-13T00:48:56.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25fa3975-c219-4b36-8260-4e9df55b1bd9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:48:56.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25fa3975-c219-4b36-8260-4e9df55b1bd9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:48:59.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37770d9a-26b5-401d-a24b-384fa5bf0487 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:48:59.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37770d9a-26b5-401d-a24b-384fa5bf0487 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@224c3b2a 2021-07-13T00:48:59.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37770d9a-26b5-401d-a24b-384fa5bf0487 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:48:59.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=37770d9a-26b5-401d-a24b-384fa5bf0487 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:49:04.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:49:04.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:49:05.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eff00390-6788-4c8e-9865-d55e909311ed PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:49:05.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eff00390-6788-4c8e-9865-d55e909311ed PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4629a57f 2021-07-13T00:49:05.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eff00390-6788-4c8e-9865-d55e909311ed PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:49:05.463Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eff00390-6788-4c8e-9865-d55e909311ed PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:49:06.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ef248b4-3953-4e06-b57d-d46fb9248a10 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:49:06.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ef248b4-3953-4e06-b57d-d46fb9248a10 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4513ba97 2021-07-13T00:49:06.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ef248b4-3953-4e06-b57d-d46fb9248a10 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:49:06.773Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ef248b4-3953-4e06-b57d-d46fb9248a10 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:49:06.922Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:49:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:49:07.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:49:09.161Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=2ef248b4-3953-4e06-b57d-d46fb9248a10 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:49:09.161Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=2ef248b4-3953-4e06-b57d-d46fb9248a10 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:49:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:49:09.367Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:49:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:49:10.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ef4e56f-67b0-4d31-85b4-08bdc1ca319d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:49:10.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ef4e56f-67b0-4d31-85b4-08bdc1ca319d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1680edee 2021-07-13T00:49:10.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ef4e56f-67b0-4d31-85b4-08bdc1ca319d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:49:10.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ef4e56f-67b0-4d31-85b4-08bdc1ca319d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:49:11.063Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=3ef4e56f-67b0-4d31-85b4-08bdc1ca319d ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:49:11.063Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=3ef4e56f-67b0-4d31-85b4-08bdc1ca319d ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:49:12.168Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=589df819-68c0-4f80-873b-efb915f157b6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:49:12.169Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=589df819-68c0-4f80-873b-efb915f157b6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@55d6017f 2021-07-13T00:49:12.169Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=589df819-68c0-4f80-873b-efb915f157b6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:49:12.169Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=589df819-68c0-4f80-873b-efb915f157b6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:49:15.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac7537f9-b8b9-4b87-8ab3-037e51224794 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:49:15.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac7537f9-b8b9-4b87-8ab3-037e51224794 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5edf54e7 2021-07-13T00:49:15.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac7537f9-b8b9-4b87-8ab3-037e51224794 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:49:15.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac7537f9-b8b9-4b87-8ab3-037e51224794 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:49:18.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1aa89e9b-c785-443e-a869-ee0bf33e372c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:49:18.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1aa89e9b-c785-443e-a869-ee0bf33e372c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6c09aa9 2021-07-13T00:49:18.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1aa89e9b-c785-443e-a869-ee0bf33e372c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:49:18.311Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1aa89e9b-c785-443e-a869-ee0bf33e372c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:49:21.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ceece14-e202-4798-bd29-1e1c4db9fd8c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:49:21.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ceece14-e202-4798-bd29-1e1c4db9fd8c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@17bef802 2021-07-13T00:49:21.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ceece14-e202-4798-bd29-1e1c4db9fd8c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:49:21.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ceece14-e202-4798-bd29-1e1c4db9fd8c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:49:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:49:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:49:24.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a56cd76d-006c-4052-8c31-7ffcfb4584fe PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:49:24.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a56cd76d-006c-4052-8c31-7ffcfb4584fe PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7286de73 2021-07-13T00:49:24.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a56cd76d-006c-4052-8c31-7ffcfb4584fe PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:49:24.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a56cd76d-006c-4052-8c31-7ffcfb4584fe PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:49:25.166Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:49:26.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4108c11-c521-41a8-973b-a609c8d0c51a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:49:26.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4108c11-c521-41a8-973b-a609c8d0c51a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@565ad4a8 2021-07-13T00:49:26.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4108c11-c521-41a8-973b-a609c8d0c51a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:49:26.957Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a4108c11-c521-41a8-973b-a609c8d0c51a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:49:29.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=849a592c-dbab-4ba5-8090-f410d1f523ff PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:49:29.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=849a592c-dbab-4ba5-8090-f410d1f523ff PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@54cd1918 2021-07-13T00:49:29.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=849a592c-dbab-4ba5-8090-f410d1f523ff PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:49:29.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=849a592c-dbab-4ba5-8090-f410d1f523ff PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:49:32.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6bb1eedf-80ec-4d15-a619-42c6d5d57fad PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:49:32.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6bb1eedf-80ec-4d15-a619-42c6d5d57fad PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3453956b 2021-07-13T00:49:32.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6bb1eedf-80ec-4d15-a619-42c6d5d57fad PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:49:32.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6bb1eedf-80ec-4d15-a619-42c6d5d57fad PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:49:35.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0c33142-e3b4-46ae-ab18-181f60ba7520 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:49:35.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0c33142-e3b4-46ae-ab18-181f60ba7520 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@70e2a0d4 2021-07-13T00:49:35.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0c33142-e3b4-46ae-ab18-181f60ba7520 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:49:35.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0c33142-e3b4-46ae-ab18-181f60ba7520 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:49:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:49:37.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:49:39.617Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02834ead-791c-4e09-a8a8-7dbff310d83e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:49:39.618Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02834ead-791c-4e09-a8a8-7dbff310d83e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@709dd5f4 2021-07-13T00:49:39.618Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02834ead-791c-4e09-a8a8-7dbff310d83e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:49:39.618Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02834ead-791c-4e09-a8a8-7dbff310d83e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:49:42.018Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4fc85034-c5df-49b5-9c6b-6370e36df6e0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:49:42.019Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4fc85034-c5df-49b5-9c6b-6370e36df6e0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4c240726 2021-07-13T00:49:42.019Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4fc85034-c5df-49b5-9c6b-6370e36df6e0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:49:42.019Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4fc85034-c5df-49b5-9c6b-6370e36df6e0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:49:45.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3cb9c6e2-d5c2-4d0b-993a-28fc99acd894 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:49:45.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3cb9c6e2-d5c2-4d0b-993a-28fc99acd894 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64742b04 2021-07-13T00:49:45.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3cb9c6e2-d5c2-4d0b-993a-28fc99acd894 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:49:45.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3cb9c6e2-d5c2-4d0b-993a-28fc99acd894 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:49:46.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:49:46.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:49:48.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3a07f76-cb43-4f13-a8fd-5ebf4077e80c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:49:48.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3a07f76-cb43-4f13-a8fd-5ebf4077e80c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@175c35e 2021-07-13T00:49:48.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3a07f76-cb43-4f13-a8fd-5ebf4077e80c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:49:48.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3a07f76-cb43-4f13-a8fd-5ebf4077e80c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:49:48.626Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=d3a07f76-cb43-4f13-a8fd-5ebf4077e80c ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:49:48.627Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=d3a07f76-cb43-4f13-a8fd-5ebf4077e80c ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:49:48.815Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:49:50.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47cde76f-d0c6-4c46-8665-237c881d668b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:49:50.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47cde76f-d0c6-4c46-8665-237c881d668b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1fd994f1 2021-07-13T00:49:50.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47cde76f-d0c6-4c46-8665-237c881d668b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:49:50.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47cde76f-d0c6-4c46-8665-237c881d668b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:49:51.462Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=47cde76f-d0c6-4c46-8665-237c881d668b ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:49:51.463Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=47cde76f-d0c6-4c46-8665-237c881d668b ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:49:52.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:49:52.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:49:53.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa7ec413-92bd-436d-bfef-768b575c7532 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:49:53.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa7ec413-92bd-436d-bfef-768b575c7532 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7328bd1b 2021-07-13T00:49:53.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa7ec413-92bd-436d-bfef-768b575c7532 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:49:53.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa7ec413-92bd-436d-bfef-768b575c7532 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:49:56.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a9241d11-d7dc-4576-86e5-54fa18113103 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:49:56.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a9241d11-d7dc-4576-86e5-54fa18113103 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@36c18471 2021-07-13T00:49:56.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a9241d11-d7dc-4576-86e5-54fa18113103 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:49:56.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a9241d11-d7dc-4576-86e5-54fa18113103 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:50:00.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02e49b8d-8f75-4f18-afc4-a94f7188d03f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:50:00.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02e49b8d-8f75-4f18-afc4-a94f7188d03f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1981dec4 2021-07-13T00:50:00.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02e49b8d-8f75-4f18-afc4-a94f7188d03f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:50:00.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02e49b8d-8f75-4f18-afc4-a94f7188d03f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:50:02.872Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c202b4a3-96a8-4982-a33e-759e871fced4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:50:02.872Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c202b4a3-96a8-4982-a33e-759e871fced4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4a2cbf19 2021-07-13T00:50:02.872Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c202b4a3-96a8-4982-a33e-759e871fced4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:50:02.872Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c202b4a3-96a8-4982-a33e-759e871fced4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:50:05.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c39a61a-d8bd-4734-ae59-5b5160453726 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:50:05.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c39a61a-d8bd-4734-ae59-5b5160453726 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7c5a0322 2021-07-13T00:50:05.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c39a61a-d8bd-4734-ae59-5b5160453726 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:50:05.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c39a61a-d8bd-4734-ae59-5b5160453726 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:50:07.917Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:50:07.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:50:09.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3163860-1a18-4b09-a846-b683081668b6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:50:09.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3163860-1a18-4b09-a846-b683081668b6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4682e2d4 2021-07-13T00:50:09.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3163860-1a18-4b09-a846-b683081668b6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:50:09.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3163860-1a18-4b09-a846-b683081668b6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:50:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:50:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:50:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:50:11.716Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc8b75cc-54b7-44c0-8895-e794936fd098 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:50:11.716Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc8b75cc-54b7-44c0-8895-e794936fd098 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@51e2633e 2021-07-13T00:50:11.716Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc8b75cc-54b7-44c0-8895-e794936fd098 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:50:11.716Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc8b75cc-54b7-44c0-8895-e794936fd098 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:50:15.022Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c78d4a3-4393-47a9-b691-b03ec49036ff PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:50:15.022Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c78d4a3-4393-47a9-b691-b03ec49036ff PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7e99ca6c 2021-07-13T00:50:15.022Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c78d4a3-4393-47a9-b691-b03ec49036ff PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:50:15.022Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4c78d4a3-4393-47a9-b691-b03ec49036ff PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:50:18.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=179b1c92-6e11-4db4-88cd-a6c0f78f0cbb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:50:18.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=179b1c92-6e11-4db4-88cd-a6c0f78f0cbb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5276858e 2021-07-13T00:50:18.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=179b1c92-6e11-4db4-88cd-a6c0f78f0cbb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:50:18.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=179b1c92-6e11-4db4-88cd-a6c0f78f0cbb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:50:21.743Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c70d1f49-20d3-459e-8e7e-3671ae044cdc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:50:21.743Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c70d1f49-20d3-459e-8e7e-3671ae044cdc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3b35956 2021-07-13T00:50:21.743Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c70d1f49-20d3-459e-8e7e-3671ae044cdc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:50:21.743Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c70d1f49-20d3-459e-8e7e-3671ae044cdc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:50:22.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:50:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:50:23.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa414c5d-fdb9-4cc1-8009-ad69b03e2f4b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:50:23.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa414c5d-fdb9-4cc1-8009-ad69b03e2f4b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@410a318d 2021-07-13T00:50:23.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa414c5d-fdb9-4cc1-8009-ad69b03e2f4b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:50:23.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa414c5d-fdb9-4cc1-8009-ad69b03e2f4b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:50:25.247Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:50:27.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f742505-3f5d-4204-8da7-bd1dc26e183e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:50:27.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f742505-3f5d-4204-8da7-bd1dc26e183e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@71f0596a 2021-07-13T00:50:27.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f742505-3f5d-4204-8da7-bd1dc26e183e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:50:27.065Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7f742505-3f5d-4204-8da7-bd1dc26e183e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:50:30.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d79d0652-9849-402b-abf4-57f79ae1e5bb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:50:30.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d79d0652-9849-402b-abf4-57f79ae1e5bb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@174cf82f 2021-07-13T00:50:30.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d79d0652-9849-402b-abf4-57f79ae1e5bb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:50:30.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d79d0652-9849-402b-abf4-57f79ae1e5bb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:50:32.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1540d83e-f010-44fd-a07a-e9c19c98050d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:50:32.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1540d83e-f010-44fd-a07a-e9c19c98050d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@eac7a79 2021-07-13T00:50:32.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1540d83e-f010-44fd-a07a-e9c19c98050d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:50:32.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1540d83e-f010-44fd-a07a-e9c19c98050d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:50:35.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7bf13e7-38e5-45f3-9c82-1f04ac60596a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:50:35.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7bf13e7-38e5-45f3-9c82-1f04ac60596a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1e3aa478 2021-07-13T00:50:35.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7bf13e7-38e5-45f3-9c82-1f04ac60596a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:50:35.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7bf13e7-38e5-45f3-9c82-1f04ac60596a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:50:37.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:50:37.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:50:38.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=23deb6af-2865-4f8e-8486-bcebe7ad7e83 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:50:38.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=23deb6af-2865-4f8e-8486-bcebe7ad7e83 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6df295dc 2021-07-13T00:50:38.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=23deb6af-2865-4f8e-8486-bcebe7ad7e83 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:50:38.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=23deb6af-2865-4f8e-8486-bcebe7ad7e83 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:50:41.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8ccc889-8b26-4bdf-8ce1-1d7af55c1c1f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:50:41.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8ccc889-8b26-4bdf-8ce1-1d7af55c1c1f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1d8f85cd 2021-07-13T00:50:41.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8ccc889-8b26-4bdf-8ce1-1d7af55c1c1f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:50:41.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f8ccc889-8b26-4bdf-8ce1-1d7af55c1c1f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:50:44.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8ae274e-77ce-4833-a73e-ba1494eefc22 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:50:44.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8ae274e-77ce-4833-a73e-ba1494eefc22 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@709d4bb7 2021-07-13T00:50:44.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8ae274e-77ce-4833-a73e-ba1494eefc22 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:50:44.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8ae274e-77ce-4833-a73e-ba1494eefc22 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:50:47.820Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7966cb1a-8634-4575-9ffb-b2d180bf9eeb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:50:47.821Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7966cb1a-8634-4575-9ffb-b2d180bf9eeb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3b4f38b8 2021-07-13T00:50:47.821Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7966cb1a-8634-4575-9ffb-b2d180bf9eeb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:50:47.821Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7966cb1a-8634-4575-9ffb-b2d180bf9eeb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:50:50.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c79f9426-9381-4475-a714-a45a0bd8c8ae PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:50:50.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c79f9426-9381-4475-a714-a45a0bd8c8ae PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@426a0678 2021-07-13T00:50:50.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c79f9426-9381-4475-a714-a45a0bd8c8ae PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:50:50.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c79f9426-9381-4475-a714-a45a0bd8c8ae PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:50:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:50:52.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:50:53.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf75b5cc-2ffe-4aed-8b6f-9850021358e5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:50:53.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf75b5cc-2ffe-4aed-8b6f-9850021358e5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@23aa5215 2021-07-13T00:50:53.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf75b5cc-2ffe-4aed-8b6f-9850021358e5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:50:53.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf75b5cc-2ffe-4aed-8b6f-9850021358e5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:50:56.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53bdcdd2-5a2c-43e1-a779-61c52e7b835b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:50:56.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53bdcdd2-5a2c-43e1-a779-61c52e7b835b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@39f07524 2021-07-13T00:50:56.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53bdcdd2-5a2c-43e1-a779-61c52e7b835b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:50:56.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=53bdcdd2-5a2c-43e1-a779-61c52e7b835b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:50:59.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=83f55581-4216-4061-ac6e-13ca0ac0b4b0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:50:59.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=83f55581-4216-4061-ac6e-13ca0ac0b4b0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7bb8005a 2021-07-13T00:50:59.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=83f55581-4216-4061-ac6e-13ca0ac0b4b0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:50:59.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=83f55581-4216-4061-ac6e-13ca0ac0b4b0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:51:02.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bd243d67-de26-4522-848f-7a7edb82cc27 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:51:02.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bd243d67-de26-4522-848f-7a7edb82cc27 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@69738093 2021-07-13T00:51:02.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bd243d67-de26-4522-848f-7a7edb82cc27 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:51:02.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bd243d67-de26-4522-848f-7a7edb82cc27 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:51:06.613Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b102376-6301-4d76-a5ce-e0dc17010948 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:51:06.613Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b102376-6301-4d76-a5ce-e0dc17010948 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@18b84d52 2021-07-13T00:51:06.613Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b102376-6301-4d76-a5ce-e0dc17010948 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:51:06.613Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b102376-6301-4d76-a5ce-e0dc17010948 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:51:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:51:07.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:51:08.778Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4bcd8a25-46d5-44f2-b6ae-31b8508176f4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:51:08.778Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4bcd8a25-46d5-44f2-b6ae-31b8508176f4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@225b5234 2021-07-13T00:51:08.778Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4bcd8a25-46d5-44f2-b6ae-31b8508176f4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:51:08.778Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4bcd8a25-46d5-44f2-b6ae-31b8508176f4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:51:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:51:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:51:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:51:11.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52e87634-1935-4ac6-8cee-91eb02c3256a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:51:11.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52e87634-1935-4ac6-8cee-91eb02c3256a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@19da3cf5 2021-07-13T00:51:11.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52e87634-1935-4ac6-8cee-91eb02c3256a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:51:11.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=52e87634-1935-4ac6-8cee-91eb02c3256a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:51:14.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c20db731-9c2e-4142-90c3-0aefbf1938ea PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:51:14.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c20db731-9c2e-4142-90c3-0aefbf1938ea PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6c585045 2021-07-13T00:51:14.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c20db731-9c2e-4142-90c3-0aefbf1938ea PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:51:14.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c20db731-9c2e-4142-90c3-0aefbf1938ea PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:51:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:51:22.920Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=142ea134-db23-48cf-96be-5110bf5b9f06 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:51:22.920Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=142ea134-db23-48cf-96be-5110bf5b9f06 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@36d04241 2021-07-13T00:51:22.920Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=142ea134-db23-48cf-96be-5110bf5b9f06 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:51:22.920Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=142ea134-db23-48cf-96be-5110bf5b9f06 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:51:22.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:51:26.230Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d1a8e58-057d-4528-929f-e24be54f7f56 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:51:26.230Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d1a8e58-057d-4528-929f-e24be54f7f56 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ee26c5d 2021-07-13T00:51:26.230Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d1a8e58-057d-4528-929f-e24be54f7f56 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:51:26.230Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d1a8e58-057d-4528-929f-e24be54f7f56 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:51:26.517Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:51:28.172Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35ab288c-cf7c-4ea6-a619-ebfeddebbe3a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:51:28.173Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35ab288c-cf7c-4ea6-a619-ebfeddebbe3a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@617cce0 2021-07-13T00:51:28.173Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35ab288c-cf7c-4ea6-a619-ebfeddebbe3a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:51:28.173Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35ab288c-cf7c-4ea6-a619-ebfeddebbe3a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:51:29.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b9f31df9-1588-41cd-bbfd-25c7bf775e8d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:51:29.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b9f31df9-1588-41cd-bbfd-25c7bf775e8d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@28e37e1f 2021-07-13T00:51:29.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b9f31df9-1588-41cd-bbfd-25c7bf775e8d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:51:29.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b9f31df9-1588-41cd-bbfd-25c7bf775e8d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:51:31.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20f4ae3d-4e0f-44e1-a5ab-001f6de3f9f3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:51:31.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20f4ae3d-4e0f-44e1-a5ab-001f6de3f9f3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c15e308 2021-07-13T00:51:31.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20f4ae3d-4e0f-44e1-a5ab-001f6de3f9f3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:51:31.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20f4ae3d-4e0f-44e1-a5ab-001f6de3f9f3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:51:33.162Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05f5ea37-8590-494b-9f22-de5e36f6be3d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:51:33.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05f5ea37-8590-494b-9f22-de5e36f6be3d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@41c8f453 2021-07-13T00:51:33.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05f5ea37-8590-494b-9f22-de5e36f6be3d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:51:33.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05f5ea37-8590-494b-9f22-de5e36f6be3d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:51:36.823Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3ab65bb-d87f-40fb-b2a2-ce3131c0ed87 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:51:36.823Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3ab65bb-d87f-40fb-b2a2-ce3131c0ed87 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@79805046 2021-07-13T00:51:36.823Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3ab65bb-d87f-40fb-b2a2-ce3131c0ed87 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:51:36.823Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3ab65bb-d87f-40fb-b2a2-ce3131c0ed87 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:51:37.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:51:37.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:51:39.128Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5529e576-4d6e-4683-ae78-6003f7edc313 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:51:39.128Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5529e576-4d6e-4683-ae78-6003f7edc313 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4a20daef 2021-07-13T00:51:39.129Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5529e576-4d6e-4683-ae78-6003f7edc313 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:51:39.129Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5529e576-4d6e-4683-ae78-6003f7edc313 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:51:42.754Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7a1fa4c-01c2-4746-8b2b-558d1a4c82df PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:51:42.755Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7a1fa4c-01c2-4746-8b2b-558d1a4c82df PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@791e7d00 2021-07-13T00:51:42.755Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7a1fa4c-01c2-4746-8b2b-558d1a4c82df PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:51:42.755Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7a1fa4c-01c2-4746-8b2b-558d1a4c82df PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:51:45.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0c07358-dd25-4d83-942b-591c40299f6f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:51:45.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0c07358-dd25-4d83-942b-591c40299f6f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7ff2741c 2021-07-13T00:51:45.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0c07358-dd25-4d83-942b-591c40299f6f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:51:45.357Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f0c07358-dd25-4d83-942b-591c40299f6f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:51:47.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7cf8136d-8b84-4b75-bacd-1fe695107c62 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:51:47.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7cf8136d-8b84-4b75-bacd-1fe695107c62 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@40b09c4 2021-07-13T00:51:47.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7cf8136d-8b84-4b75-bacd-1fe695107c62 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:51:47.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7cf8136d-8b84-4b75-bacd-1fe695107c62 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:51:52.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:51:52.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:51:52.917Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:51:52.925Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:51:55.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:51:57.221Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76c97e00-3c57-4d06-9be6-b925a42d17df PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:51:57.221Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76c97e00-3c57-4d06-9be6-b925a42d17df PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@48b05c08 2021-07-13T00:51:57.221Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76c97e00-3c57-4d06-9be6-b925a42d17df PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:51:57.221Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=76c97e00-3c57-4d06-9be6-b925a42d17df PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:51:57.660Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:51:58.526Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac96c38e-4a9d-49b6-9365-891975bc3d9a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:51:58.526Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac96c38e-4a9d-49b6-9365-891975bc3d9a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@42ccde63 2021-07-13T00:51:58.526Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac96c38e-4a9d-49b6-9365-891975bc3d9a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:51:58.526Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ac96c38e-4a9d-49b6-9365-891975bc3d9a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:51:59.062Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=ac96c38e-4a9d-49b6-9365-891975bc3d9a ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:51:59.062Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=ac96c38e-4a9d-49b6-9365-891975bc3d9a ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:52:00.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05de842e-c25a-4389-9721-2cd1b5ba5223 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:52:00.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05de842e-c25a-4389-9721-2cd1b5ba5223 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ef200b2 2021-07-13T00:52:00.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05de842e-c25a-4389-9721-2cd1b5ba5223 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:52:00.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05de842e-c25a-4389-9721-2cd1b5ba5223 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:52:00.662Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=05de842e-c25a-4389-9721-2cd1b5ba5223 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:52:00.662Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=05de842e-c25a-4389-9721-2cd1b5ba5223 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:52:01.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26980bc8-8fb8-4b37-a06d-32006fa66b2b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:52:01.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26980bc8-8fb8-4b37-a06d-32006fa66b2b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@72eb7981 2021-07-13T00:52:01.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26980bc8-8fb8-4b37-a06d-32006fa66b2b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:52:01.857Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26980bc8-8fb8-4b37-a06d-32006fa66b2b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:52:03.657Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=80228543-28a0-4eca-a84b-5aefdebc471b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:52:03.657Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=80228543-28a0-4eca-a84b-5aefdebc471b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@38712bc8 2021-07-13T00:52:03.657Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=80228543-28a0-4eca-a84b-5aefdebc471b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:52:03.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=80228543-28a0-4eca-a84b-5aefdebc471b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:52:06.167Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e51d74d5-4a50-4655-8842-66c7403f09f8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:52:06.167Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e51d74d5-4a50-4655-8842-66c7403f09f8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1cead563 2021-07-13T00:52:06.167Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e51d74d5-4a50-4655-8842-66c7403f09f8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:52:06.167Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e51d74d5-4a50-4655-8842-66c7403f09f8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:52:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:52:07.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:52:09.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be009cb7-3ae0-4e3e-b88e-bd5fced57ab1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:52:09.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be009cb7-3ae0-4e3e-b88e-bd5fced57ab1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2223a1dc 2021-07-13T00:52:09.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be009cb7-3ae0-4e3e-b88e-bd5fced57ab1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:52:09.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be009cb7-3ae0-4e3e-b88e-bd5fced57ab1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:52:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:52:09.369Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:52:09.369Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:52:11.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2134912a-73b9-45b0-a3c8-876713c69926 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:52:11.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2134912a-73b9-45b0-a3c8-876713c69926 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@bade368 2021-07-13T00:52:11.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2134912a-73b9-45b0-a3c8-876713c69926 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:52:11.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2134912a-73b9-45b0-a3c8-876713c69926 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:52:15.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a6c5f63-e204-49f4-aac3-e65f36c57c9c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:52:15.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a6c5f63-e204-49f4-aac3-e65f36c57c9c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3a33a6be 2021-07-13T00:52:15.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a6c5f63-e204-49f4-aac3-e65f36c57c9c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:52:15.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a6c5f63-e204-49f4-aac3-e65f36c57c9c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:52:18.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9283a1f4-2b61-4e10-8b4c-38308506655b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:52:18.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9283a1f4-2b61-4e10-8b4c-38308506655b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@70b83d56 2021-07-13T00:52:18.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9283a1f4-2b61-4e10-8b4c-38308506655b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:52:18.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9283a1f4-2b61-4e10-8b4c-38308506655b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:52:20.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6b5b13a-4d27-4d87-912a-36605622897e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:52:20.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6b5b13a-4d27-4d87-912a-36605622897e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4f2a5fdc 2021-07-13T00:52:20.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6b5b13a-4d27-4d87-912a-36605622897e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:52:20.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6b5b13a-4d27-4d87-912a-36605622897e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:52:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:52:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:52:23.971Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=f6b5b13a-4d27-4d87-912a-36605622897e ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:52:23.972Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=f6b5b13a-4d27-4d87-912a-36605622897e ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:52:23.972Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f6b5b13a-4d27-4d87-912a-36605622897e ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:52:23.972Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f6b5b13a-4d27-4d87-912a-36605622897e ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:52:25.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc0fd3eb-5d09-400a-8d74-11fbb8746c2c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:52:25.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc0fd3eb-5d09-400a-8d74-11fbb8746c2c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2bf853f2 2021-07-13T00:52:25.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc0fd3eb-5d09-400a-8d74-11fbb8746c2c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:52:25.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bc0fd3eb-5d09-400a-8d74-11fbb8746c2c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:52:26.560Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=bc0fd3eb-5d09-400a-8d74-11fbb8746c2c ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:52:26.560Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=bc0fd3eb-5d09-400a-8d74-11fbb8746c2c ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:52:27.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6b493d77-0d96-4d3a-b229-bc5f6148a000 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:52:27.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6b493d77-0d96-4d3a-b229-bc5f6148a000 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3d3ad825 2021-07-13T00:52:27.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6b493d77-0d96-4d3a-b229-bc5f6148a000 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:52:27.315Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6b493d77-0d96-4d3a-b229-bc5f6148a000 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:52:27.761Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:52:29.913Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46ff0538-7cbb-4f36-93ca-59f0818a77a9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:52:29.913Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46ff0538-7cbb-4f36-93ca-59f0818a77a9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2073b4e1 2021-07-13T00:52:29.913Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46ff0538-7cbb-4f36-93ca-59f0818a77a9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:52:29.913Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=46ff0538-7cbb-4f36-93ca-59f0818a77a9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:52:33.017Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8587ca8d-0e2d-4411-8657-ddd9a2626b76 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:52:33.017Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8587ca8d-0e2d-4411-8657-ddd9a2626b76 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@497d04de 2021-07-13T00:52:33.017Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8587ca8d-0e2d-4411-8657-ddd9a2626b76 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:52:33.017Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8587ca8d-0e2d-4411-8657-ddd9a2626b76 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:52:35.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7b72100-a165-4e4d-a503-e262cb8cb384 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:52:35.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7b72100-a165-4e4d-a503-e262cb8cb384 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@d5349aa 2021-07-13T00:52:35.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7b72100-a165-4e4d-a503-e262cb8cb384 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:52:35.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7b72100-a165-4e4d-a503-e262cb8cb384 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:52:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:52:37.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:52:38.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d734665-7162-4a8d-bc11-ff0786c6c387 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:52:38.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d734665-7162-4a8d-bc11-ff0786c6c387 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@664353d5 2021-07-13T00:52:38.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d734665-7162-4a8d-bc11-ff0786c6c387 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:52:38.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d734665-7162-4a8d-bc11-ff0786c6c387 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:52:41.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a9d8fd47-fb1c-41a5-abb0-4d4c74ab59a7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:52:41.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a9d8fd47-fb1c-41a5-abb0-4d4c74ab59a7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5baa3740 2021-07-13T00:52:41.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a9d8fd47-fb1c-41a5-abb0-4d4c74ab59a7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:52:41.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a9d8fd47-fb1c-41a5-abb0-4d4c74ab59a7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:52:44.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f048d3ac-bc82-4296-870f-3bc2d87350ec PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:52:44.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f048d3ac-bc82-4296-870f-3bc2d87350ec PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7ad0931f 2021-07-13T00:52:44.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f048d3ac-bc82-4296-870f-3bc2d87350ec PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:52:44.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f048d3ac-bc82-4296-870f-3bc2d87350ec PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:52:47.882Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6779b47-58df-4922-96ee-a702a424e10b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:52:47.882Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6779b47-58df-4922-96ee-a702a424e10b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6dd99bf6 2021-07-13T00:52:47.882Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6779b47-58df-4922-96ee-a702a424e10b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:52:47.882Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c6779b47-58df-4922-96ee-a702a424e10b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:52:51.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=897e9cce-854e-4959-a78d-0a2aee5e64a5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:52:51.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=897e9cce-854e-4959-a78d-0a2aee5e64a5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1178b063 2021-07-13T00:52:51.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=897e9cce-854e-4959-a78d-0a2aee5e64a5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:52:51.166Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=897e9cce-854e-4959-a78d-0a2aee5e64a5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:52:52.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:52:52.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:52:53.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0fc62275-ce28-4ce1-8803-52755f2275cd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:52:53.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0fc62275-ce28-4ce1-8803-52755f2275cd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b62953b 2021-07-13T00:52:53.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0fc62275-ce28-4ce1-8803-52755f2275cd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:52:53.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0fc62275-ce28-4ce1-8803-52755f2275cd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:52:56.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7ca0417-6363-4bf7-a29e-72ae8b6fdf8d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:52:56.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7ca0417-6363-4bf7-a29e-72ae8b6fdf8d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@65d4b523 2021-07-13T00:52:56.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7ca0417-6363-4bf7-a29e-72ae8b6fdf8d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:52:56.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7ca0417-6363-4bf7-a29e-72ae8b6fdf8d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:52:59.575Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6bdd5e42-fdda-44df-9868-963ee226411d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:52:59.575Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6bdd5e42-fdda-44df-9868-963ee226411d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@44738444 2021-07-13T00:52:59.575Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6bdd5e42-fdda-44df-9868-963ee226411d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:52:59.575Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6bdd5e42-fdda-44df-9868-963ee226411d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:53:02.972Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=855aacb8-ecc3-4372-a70e-cbe4620fad28 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:53:02.972Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=855aacb8-ecc3-4372-a70e-cbe4620fad28 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@105cdf01 2021-07-13T00:53:02.972Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=855aacb8-ecc3-4372-a70e-cbe4620fad28 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:53:02.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=855aacb8-ecc3-4372-a70e-cbe4620fad28 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:53:05.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c044b85-d036-47ef-bc27-aa07ce9b264c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:53:05.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c044b85-d036-47ef-bc27-aa07ce9b264c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f984b7c 2021-07-13T00:53:05.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c044b85-d036-47ef-bc27-aa07ce9b264c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:53:05.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3c044b85-d036-47ef-bc27-aa07ce9b264c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:53:07.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:53:07.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:53:09.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1c6e9c8e-67a7-48b7-b204-3e1b898d72ae PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:53:09.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1c6e9c8e-67a7-48b7-b204-3e1b898d72ae PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5fa58801 2021-07-13T00:53:09.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1c6e9c8e-67a7-48b7-b204-3e1b898d72ae PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:53:09.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1c6e9c8e-67a7-48b7-b204-3e1b898d72ae PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:53:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:53:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:53:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:53:11.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dba1d2fd-85dd-4641-9d40-c2602fcb0b82 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:53:11.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dba1d2fd-85dd-4641-9d40-c2602fcb0b82 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@72c6615 2021-07-13T00:53:11.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dba1d2fd-85dd-4641-9d40-c2602fcb0b82 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:53:11.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dba1d2fd-85dd-4641-9d40-c2602fcb0b82 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:53:15.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=820513d2-09d7-49b6-9550-b38537688092 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:53:15.461Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=820513d2-09d7-49b6-9550-b38537688092 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@71fe1526 2021-07-13T00:53:15.461Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=820513d2-09d7-49b6-9550-b38537688092 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:53:15.461Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=820513d2-09d7-49b6-9550-b38537688092 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:53:17.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f34957e4-fb37-4f14-bbe1-b1d2f4b24f1c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:53:17.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f34957e4-fb37-4f14-bbe1-b1d2f4b24f1c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2c6073ce 2021-07-13T00:53:17.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f34957e4-fb37-4f14-bbe1-b1d2f4b24f1c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:53:17.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f34957e4-fb37-4f14-bbe1-b1d2f4b24f1c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:53:20.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=58e60fa0-e337-441d-b13b-0e7c118065ad PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:53:20.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=58e60fa0-e337-441d-b13b-0e7c118065ad PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f7613c3 2021-07-13T00:53:20.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=58e60fa0-e337-441d-b13b-0e7c118065ad PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:53:20.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=58e60fa0-e337-441d-b13b-0e7c118065ad PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:53:22.918Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:53:22.929Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:53:23.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad7ce6e6-5eef-4fa8-960a-6f6f24b823d1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:53:23.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad7ce6e6-5eef-4fa8-960a-6f6f24b823d1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3b0edaf6 2021-07-13T00:53:23.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad7ce6e6-5eef-4fa8-960a-6f6f24b823d1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:53:23.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ad7ce6e6-5eef-4fa8-960a-6f6f24b823d1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:53:25.035Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:53:27.030Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=507ab6d6-b576-4471-9a1a-b710b33bb280 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:53:27.030Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=507ab6d6-b576-4471-9a1a-b710b33bb280 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3ad8bf94 2021-07-13T00:53:27.030Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=507ab6d6-b576-4471-9a1a-b710b33bb280 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:53:27.030Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=507ab6d6-b576-4471-9a1a-b710b33bb280 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:53:29.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ebb912e7-3d4b-4a0f-85f1-69ecb0056b64 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:53:29.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ebb912e7-3d4b-4a0f-85f1-69ecb0056b64 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@15094502 2021-07-13T00:53:29.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ebb912e7-3d4b-4a0f-85f1-69ecb0056b64 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:53:29.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ebb912e7-3d4b-4a0f-85f1-69ecb0056b64 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:53:32.610Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2f3e907-7f0e-47fc-8135-59ca2ef4a95f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:53:32.610Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2f3e907-7f0e-47fc-8135-59ca2ef4a95f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7a8ea46b 2021-07-13T00:53:32.610Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2f3e907-7f0e-47fc-8135-59ca2ef4a95f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:53:32.610Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2f3e907-7f0e-47fc-8135-59ca2ef4a95f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:53:36.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1432496-1282-45de-8c6e-eeac80de9bd7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:53:36.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1432496-1282-45de-8c6e-eeac80de9bd7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@379a18a6 2021-07-13T00:53:36.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1432496-1282-45de-8c6e-eeac80de9bd7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:53:36.416Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1432496-1282-45de-8c6e-eeac80de9bd7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:53:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:53:37.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:53:38.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6baa5290-d73a-48c8-9069-aeb838948bf3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:53:38.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6baa5290-d73a-48c8-9069-aeb838948bf3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@484135de 2021-07-13T00:53:38.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6baa5290-d73a-48c8-9069-aeb838948bf3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:53:38.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6baa5290-d73a-48c8-9069-aeb838948bf3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:53:41.461Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5eb184cb-0196-4de7-8c37-938f214ac61f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:53:41.461Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5eb184cb-0196-4de7-8c37-938f214ac61f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@173f4881 2021-07-13T00:53:41.461Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5eb184cb-0196-4de7-8c37-938f214ac61f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:53:41.461Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5eb184cb-0196-4de7-8c37-938f214ac61f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:53:45.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01252f30-d743-49a8-8710-8b0d2c1fb0e3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:53:45.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01252f30-d743-49a8-8710-8b0d2c1fb0e3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@189d1bf1 2021-07-13T00:53:45.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01252f30-d743-49a8-8710-8b0d2c1fb0e3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:53:45.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01252f30-d743-49a8-8710-8b0d2c1fb0e3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:53:48.014Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ef559347-03fe-43dd-b659-b167e04da9e2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:53:48.014Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ef559347-03fe-43dd-b659-b167e04da9e2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7fa56f6c 2021-07-13T00:53:48.014Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ef559347-03fe-43dd-b659-b167e04da9e2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:53:48.014Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ef559347-03fe-43dd-b659-b167e04da9e2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:53:50.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae1f6b49-b123-4dff-a171-e79fb93516dc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:53:50.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae1f6b49-b123-4dff-a171-e79fb93516dc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3e0b4a83 2021-07-13T00:53:50.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae1f6b49-b123-4dff-a171-e79fb93516dc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:53:50.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae1f6b49-b123-4dff-a171-e79fb93516dc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:53:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:53:52.929Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:53:53.670Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e572073d-da83-4540-8518-d0a6808671fa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:53:53.670Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e572073d-da83-4540-8518-d0a6808671fa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@539d7683 2021-07-13T00:53:53.670Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e572073d-da83-4540-8518-d0a6808671fa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:53:53.670Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e572073d-da83-4540-8518-d0a6808671fa PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:53:56.871Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a0930a01-d748-458f-a501-94063e970273 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:53:56.871Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a0930a01-d748-458f-a501-94063e970273 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@232fbd92 2021-07-13T00:53:56.871Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a0930a01-d748-458f-a501-94063e970273 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:53:56.871Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a0930a01-d748-458f-a501-94063e970273 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:53:59.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e535d852-95af-487c-965e-4fe7d1d5c257 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:53:59.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e535d852-95af-487c-965e-4fe7d1d5c257 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@77a56894 2021-07-13T00:53:59.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e535d852-95af-487c-965e-4fe7d1d5c257 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:53:59.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e535d852-95af-487c-965e-4fe7d1d5c257 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:54:02.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf04b888-984e-4015-9d07-2191cfd3455c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:54:02.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf04b888-984e-4015-9d07-2191cfd3455c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@139e6bb1 2021-07-13T00:54:02.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf04b888-984e-4015-9d07-2191cfd3455c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:54:02.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cf04b888-984e-4015-9d07-2191cfd3455c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:54:05.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8cba108a-2c36-4e00-8c3d-77fd98362148 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:54:05.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8cba108a-2c36-4e00-8c3d-77fd98362148 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2b8f2e2f 2021-07-13T00:54:05.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8cba108a-2c36-4e00-8c3d-77fd98362148 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:54:05.865Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8cba108a-2c36-4e00-8c3d-77fd98362148 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:54:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:54:07.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:54:08.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2704d1c-eebe-4f8d-9a03-452fb149300c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:54:08.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2704d1c-eebe-4f8d-9a03-452fb149300c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7512c5e1 2021-07-13T00:54:08.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2704d1c-eebe-4f8d-9a03-452fb149300c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:54:08.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2704d1c-eebe-4f8d-9a03-452fb149300c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:54:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:54:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:54:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:54:12.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1515e96-3b05-4afc-83c2-80c6230f11fe PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:54:12.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1515e96-3b05-4afc-83c2-80c6230f11fe PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@29afe21f 2021-07-13T00:54:12.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1515e96-3b05-4afc-83c2-80c6230f11fe PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:54:12.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1515e96-3b05-4afc-83c2-80c6230f11fe PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:54:14.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3df7f6b8-8ea8-4e28-8a41-510e6f492ee8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:54:14.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3df7f6b8-8ea8-4e28-8a41-510e6f492ee8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@498cc778 2021-07-13T00:54:14.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3df7f6b8-8ea8-4e28-8a41-510e6f492ee8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:54:14.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3df7f6b8-8ea8-4e28-8a41-510e6f492ee8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:54:18.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8eafc4db-15ea-4be2-9179-6bef29ab48f7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:54:18.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8eafc4db-15ea-4be2-9179-6bef29ab48f7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b207630 2021-07-13T00:54:18.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8eafc4db-15ea-4be2-9179-6bef29ab48f7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:54:18.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8eafc4db-15ea-4be2-9179-6bef29ab48f7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:54:20.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e48f7016-c61e-470a-a6ff-8e6e23620af6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:54:20.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e48f7016-c61e-470a-a6ff-8e6e23620af6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78fe9578 2021-07-13T00:54:20.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e48f7016-c61e-470a-a6ff-8e6e23620af6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:54:20.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e48f7016-c61e-470a-a6ff-8e6e23620af6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:54:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:54:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:54:23.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5769c390-33a6-4bc8-aed0-7b5834ceb074 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:54:23.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5769c390-33a6-4bc8-aed0-7b5834ceb074 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2e173f62 2021-07-13T00:54:23.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5769c390-33a6-4bc8-aed0-7b5834ceb074 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:54:23.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5769c390-33a6-4bc8-aed0-7b5834ceb074 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:54:27.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40bf6a1c-8d5f-4c1e-b9ea-20d7f0f07e13 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:54:27.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40bf6a1c-8d5f-4c1e-b9ea-20d7f0f07e13 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@10f8adf 2021-07-13T00:54:27.213Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40bf6a1c-8d5f-4c1e-b9ea-20d7f0f07e13 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:54:27.214Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40bf6a1c-8d5f-4c1e-b9ea-20d7f0f07e13 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:54:29.261Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:54:29.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b053d4e5-8c57-409b-870c-c5f3b37437b8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:54:29.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b053d4e5-8c57-409b-870c-c5f3b37437b8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1906fd38 2021-07-13T00:54:29.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b053d4e5-8c57-409b-870c-c5f3b37437b8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:54:29.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b053d4e5-8c57-409b-870c-c5f3b37437b8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:54:33.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba9a8eb5-be06-4d6c-96a3-c27b8e7db108 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:54:33.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba9a8eb5-be06-4d6c-96a3-c27b8e7db108 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@15dffe35 2021-07-13T00:54:33.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba9a8eb5-be06-4d6c-96a3-c27b8e7db108 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:54:33.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba9a8eb5-be06-4d6c-96a3-c27b8e7db108 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:54:35.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f5b438f-7216-4325-8cfd-b73d49deb85c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:54:35.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f5b438f-7216-4325-8cfd-b73d49deb85c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2a42fc3 2021-07-13T00:54:35.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f5b438f-7216-4325-8cfd-b73d49deb85c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:54:35.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9f5b438f-7216-4325-8cfd-b73d49deb85c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:54:37.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:54:37.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:54:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:54:37.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:54:38.673Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=9f5b438f-7216-4325-8cfd-b73d49deb85c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:54:38.674Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=9f5b438f-7216-4325-8cfd-b73d49deb85c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:54:38.675Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9f5b438f-7216-4325-8cfd-b73d49deb85c ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:54:38.675Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9f5b438f-7216-4325-8cfd-b73d49deb85c ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:54:39.711Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:54:39.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5db63581-d7ae-4b30-a7a0-977335393a34 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:54:39.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5db63581-d7ae-4b30-a7a0-977335393a34 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f00043a 2021-07-13T00:54:39.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5db63581-d7ae-4b30-a7a0-977335393a34 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:54:39.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5db63581-d7ae-4b30-a7a0-977335393a34 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:54:41.063Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=5db63581-d7ae-4b30-a7a0-977335393a34 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:54:41.063Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=5db63581-d7ae-4b30-a7a0-977335393a34 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:54:42.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2fa13212-59fb-4542-971a-d6eee38b5c05 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:54:42.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2fa13212-59fb-4542-971a-d6eee38b5c05 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3b85ab84 2021-07-13T00:54:42.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2fa13212-59fb-4542-971a-d6eee38b5c05 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:54:42.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2fa13212-59fb-4542-971a-d6eee38b5c05 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:54:43.763Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=2fa13212-59fb-4542-971a-d6eee38b5c05 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:54:43.763Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=2fa13212-59fb-4542-971a-d6eee38b5c05 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:54:45.085Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2c7d189-5ec7-4b69-9cf0-50265f9be830 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:54:45.085Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2c7d189-5ec7-4b69-9cf0-50265f9be830 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@187c5100 2021-07-13T00:54:45.085Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2c7d189-5ec7-4b69-9cf0-50265f9be830 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:54:45.085Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2c7d189-5ec7-4b69-9cf0-50265f9be830 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:54:48.915Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb8dd836-657f-495c-868e-1d3b0646d11b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:54:48.915Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb8dd836-657f-495c-868e-1d3b0646d11b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@67a2b7e1 2021-07-13T00:54:48.915Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb8dd836-657f-495c-868e-1d3b0646d11b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:54:48.915Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cb8dd836-657f-495c-868e-1d3b0646d11b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:54:51.437Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c124c02-131f-43ab-a35c-56cce0469af4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:54:51.438Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c124c02-131f-43ab-a35c-56cce0469af4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@38f205fb 2021-07-13T00:54:51.438Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c124c02-131f-43ab-a35c-56cce0469af4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:54:51.438Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9c124c02-131f-43ab-a35c-56cce0469af4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:54:52.917Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:54:52.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:54:53.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7fe87f6-613b-4349-bb51-cc9f4ce70991 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:54:53.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7fe87f6-613b-4349-bb51-cc9f4ce70991 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2ebd40ea 2021-07-13T00:54:53.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7fe87f6-613b-4349-bb51-cc9f4ce70991 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:54:53.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f7fe87f6-613b-4349-bb51-cc9f4ce70991 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:54:57.414Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32ac314e-17f1-44a0-b0c4-6bdbb203bbf6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:54:57.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32ac314e-17f1-44a0-b0c4-6bdbb203bbf6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5433e06c 2021-07-13T00:54:57.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32ac314e-17f1-44a0-b0c4-6bdbb203bbf6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:54:57.415Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32ac314e-17f1-44a0-b0c4-6bdbb203bbf6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:55:00.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a75d5e6-5432-4f8f-a91c-5333ec849065 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:55:00.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a75d5e6-5432-4f8f-a91c-5333ec849065 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@265c48b4 2021-07-13T00:55:00.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a75d5e6-5432-4f8f-a91c-5333ec849065 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:55:00.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2a75d5e6-5432-4f8f-a91c-5333ec849065 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:55:02.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b2ac910-e42c-4d78-9351-7a8a48bf1086 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:55:02.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b2ac910-e42c-4d78-9351-7a8a48bf1086 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@ea94c59 2021-07-13T00:55:02.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b2ac910-e42c-4d78-9351-7a8a48bf1086 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:55:02.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b2ac910-e42c-4d78-9351-7a8a48bf1086 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:55:06.320Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=793d8a9c-720b-4ad9-8c25-da914eac8812 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:55:06.320Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=793d8a9c-720b-4ad9-8c25-da914eac8812 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ad7e776 2021-07-13T00:55:06.320Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=793d8a9c-720b-4ad9-8c25-da914eac8812 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:55:06.320Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=793d8a9c-720b-4ad9-8c25-da914eac8812 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:55:07.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:55:07.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:55:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:55:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:55:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:55:10.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:55:10.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:55:13.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:55:16.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:55:19.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:55:22.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:55:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:55:22.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:55:25.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:55:26.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abc7969b-0d95-4305-af7b-9e9d8f8ed509 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:55:26.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abc7969b-0d95-4305-af7b-9e9d8f8ed509 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@12c4ec0 2021-07-13T00:55:26.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abc7969b-0d95-4305-af7b-9e9d8f8ed509 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:55:26.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abc7969b-0d95-4305-af7b-9e9d8f8ed509 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:55:28.310Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:55:28.321Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b06eaa8-99ae-4bf6-a548-0b92cee92046 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:55:28.321Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b06eaa8-99ae-4bf6-a548-0b92cee92046 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@45f1bfb9 2021-07-13T00:55:28.322Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b06eaa8-99ae-4bf6-a548-0b92cee92046 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:55:28.322Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4b06eaa8-99ae-4bf6-a548-0b92cee92046 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:55:28.700Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:55:31.325Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=4b06eaa8-99ae-4bf6-a548-0b92cee92046 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:55:31.326Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=4b06eaa8-99ae-4bf6-a548-0b92cee92046 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:55:31.326Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=4b06eaa8-99ae-4bf6-a548-0b92cee92046 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:55:31.326Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=4b06eaa8-99ae-4bf6-a548-0b92cee92046 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:55:32.271Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fd88b6e-b7af-4a7c-aea9-cf8b0b492a1a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:55:32.271Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fd88b6e-b7af-4a7c-aea9-cf8b0b492a1a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@17ee8da 2021-07-13T00:55:32.271Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fd88b6e-b7af-4a7c-aea9-cf8b0b492a1a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:55:32.271Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6fd88b6e-b7af-4a7c-aea9-cf8b0b492a1a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:55:34.363Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6fd88b6e-b7af-4a7c-aea9-cf8b0b492a1a ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:55:34.363Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6fd88b6e-b7af-4a7c-aea9-cf8b0b492a1a ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:55:35.672Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df1bbe0e-7c21-4a05-9f57-2f494b7d85c5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:55:35.672Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df1bbe0e-7c21-4a05-9f57-2f494b7d85c5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@17de49c6 2021-07-13T00:55:35.672Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df1bbe0e-7c21-4a05-9f57-2f494b7d85c5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:55:35.672Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df1bbe0e-7c21-4a05-9f57-2f494b7d85c5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:55:37.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9135723e-9a5c-47b5-98b5-b07927d928e4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:55:37.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9135723e-9a5c-47b5-98b5-b07927d928e4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f9a20f3 2021-07-13T00:55:37.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9135723e-9a5c-47b5-98b5-b07927d928e4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:55:37.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9135723e-9a5c-47b5-98b5-b07927d928e4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:55:37.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:55:37.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:55:39.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21acd3c2-e15d-407b-b22a-2be9c39cec52 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:55:39.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21acd3c2-e15d-407b-b22a-2be9c39cec52 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@29d43be2 2021-07-13T00:55:39.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21acd3c2-e15d-407b-b22a-2be9c39cec52 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:55:39.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21acd3c2-e15d-407b-b22a-2be9c39cec52 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:55:41.618Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a38d6740-20c8-41b0-a6ab-9db4738a7be0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:55:41.618Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a38d6740-20c8-41b0-a6ab-9db4738a7be0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@67e334ef 2021-07-13T00:55:41.618Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a38d6740-20c8-41b0-a6ab-9db4738a7be0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:55:41.618Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a38d6740-20c8-41b0-a6ab-9db4738a7be0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:55:43.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=541a52e2-e6f1-457c-a8d2-6360e52fb4ac PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:55:43.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=541a52e2-e6f1-457c-a8d2-6360e52fb4ac PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@216d1ac 2021-07-13T00:55:43.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=541a52e2-e6f1-457c-a8d2-6360e52fb4ac PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:55:43.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=541a52e2-e6f1-457c-a8d2-6360e52fb4ac PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:55:44.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01009c7e-57d8-4705-8c63-a02c5f3669f9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:55:44.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01009c7e-57d8-4705-8c63-a02c5f3669f9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@c85f4a5 2021-07-13T00:55:44.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01009c7e-57d8-4705-8c63-a02c5f3669f9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:55:44.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01009c7e-57d8-4705-8c63-a02c5f3669f9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:55:46.717Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7677729-9b2e-44b3-b4b5-e402979daf5e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:55:46.717Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7677729-9b2e-44b3-b4b5-e402979daf5e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@486d5fc 2021-07-13T00:55:46.717Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7677729-9b2e-44b3-b4b5-e402979daf5e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:55:46.717Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7677729-9b2e-44b3-b4b5-e402979daf5e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:55:48.369Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d200c458-0b46-464f-989c-6b970b92e178 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:55:48.369Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d200c458-0b46-464f-989c-6b970b92e178 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7abcddcf 2021-07-13T00:55:48.369Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d200c458-0b46-464f-989c-6b970b92e178 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:55:48.369Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d200c458-0b46-464f-989c-6b970b92e178 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:55:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:55:52.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:55:53.722Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78a57db5-bc69-4087-bbbe-ee6df935f633 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:55:53.722Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78a57db5-bc69-4087-bbbe-ee6df935f633 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4e93f885 2021-07-13T00:55:53.722Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78a57db5-bc69-4087-bbbe-ee6df935f633 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:55:53.722Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=78a57db5-bc69-4087-bbbe-ee6df935f633 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:55:55.562Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6bdc674b-194d-49ca-b3b5-7817a0165d48 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:55:55.562Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6bdc674b-194d-49ca-b3b5-7817a0165d48 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6d901d2e 2021-07-13T00:55:55.562Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6bdc674b-194d-49ca-b3b5-7817a0165d48 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:55:55.562Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6bdc674b-194d-49ca-b3b5-7817a0165d48 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:55:56.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba279b96-46aa-40e5-bfc7-0c397117c7da PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:55:56.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba279b96-46aa-40e5-bfc7-0c397117c7da PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f8b1c55 2021-07-13T00:55:56.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba279b96-46aa-40e5-bfc7-0c397117c7da PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:55:56.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ba279b96-46aa-40e5-bfc7-0c397117c7da PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:55:58.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11af9e8e-8b62-46cc-b901-de990c003421 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:55:58.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11af9e8e-8b62-46cc-b901-de990c003421 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4345f0ce 2021-07-13T00:55:58.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11af9e8e-8b62-46cc-b901-de990c003421 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:55:58.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11af9e8e-8b62-46cc-b901-de990c003421 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:56:00.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0dfec2e3-f572-431e-b5d8-3364ac80d2fa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:56:00.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0dfec2e3-f572-431e-b5d8-3364ac80d2fa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@12437ff3 2021-07-13T00:56:00.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0dfec2e3-f572-431e-b5d8-3364ac80d2fa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:56:00.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0dfec2e3-f572-431e-b5d8-3364ac80d2fa PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:56:01.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=294c2e20-b476-448b-955f-bfd9c1741a04 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:56:01.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=294c2e20-b476-448b-955f-bfd9c1741a04 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f5769f8 2021-07-13T00:56:01.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=294c2e20-b476-448b-955f-bfd9c1741a04 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:56:01.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=294c2e20-b476-448b-955f-bfd9c1741a04 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:56:03.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0123bfe1-08dc-485e-a2ff-94b1959bb3d0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:56:03.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0123bfe1-08dc-485e-a2ff-94b1959bb3d0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78b9bbf3 2021-07-13T00:56:03.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0123bfe1-08dc-485e-a2ff-94b1959bb3d0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:56:03.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0123bfe1-08dc-485e-a2ff-94b1959bb3d0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:56:04.567Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d02f43dc-bf9f-4c07-b8db-73cd56875ba4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:56:04.567Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d02f43dc-bf9f-4c07-b8db-73cd56875ba4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1416e7aa 2021-07-13T00:56:04.567Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d02f43dc-bf9f-4c07-b8db-73cd56875ba4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:56:04.567Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d02f43dc-bf9f-4c07-b8db-73cd56875ba4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:56:05.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6b1ce987-56ba-450b-aac3-0b3ea4bc9f90 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:56:05.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6b1ce987-56ba-450b-aac3-0b3ea4bc9f90 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1da3ae3c 2021-07-13T00:56:05.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6b1ce987-56ba-450b-aac3-0b3ea4bc9f90 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:56:05.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6b1ce987-56ba-450b-aac3-0b3ea4bc9f90 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:56:07.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:56:07.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:56:07.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:56:07.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:56:08.964Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=6b1ce987-56ba-450b-aac3-0b3ea4bc9f90 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:56:08.964Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=6b1ce987-56ba-450b-aac3-0b3ea4bc9f90 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:56:08.965Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6b1ce987-56ba-450b-aac3-0b3ea4bc9f90 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:56:08.965Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6b1ce987-56ba-450b-aac3-0b3ea4bc9f90 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:56:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:56:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:56:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:56:10.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:56:11.575Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f13ac23b-48e7-4aa4-a9a1-03e75b24c24c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:56:11.575Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f13ac23b-48e7-4aa4-a9a1-03e75b24c24c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@597f0cd9 2021-07-13T00:56:11.575Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f13ac23b-48e7-4aa4-a9a1-03e75b24c24c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:56:11.575Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f13ac23b-48e7-4aa4-a9a1-03e75b24c24c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:56:12.472Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f13ac23b-48e7-4aa4-a9a1-03e75b24c24c ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:56:12.472Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=f13ac23b-48e7-4aa4-a9a1-03e75b24c24c ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:56:12.763Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:56:16.619Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a74ecb41-9b98-44df-a891-d91c90d17312 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:56:16.619Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a74ecb41-9b98-44df-a891-d91c90d17312 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4f24d3be 2021-07-13T00:56:16.620Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a74ecb41-9b98-44df-a891-d91c90d17312 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:56:16.620Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a74ecb41-9b98-44df-a891-d91c90d17312 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:56:18.823Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9144f415-d1e1-4355-97fc-129342809d59 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:56:18.823Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9144f415-d1e1-4355-97fc-129342809d59 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4f764628 2021-07-13T00:56:18.823Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9144f415-d1e1-4355-97fc-129342809d59 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:56:18.823Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9144f415-d1e1-4355-97fc-129342809d59 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:56:21.829Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=9144f415-d1e1-4355-97fc-129342809d59 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:56:21.829Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=9144f415-d1e1-4355-97fc-129342809d59 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:56:21.829Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9144f415-d1e1-4355-97fc-129342809d59 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:56:21.829Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9144f415-d1e1-4355-97fc-129342809d59 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:56:22.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:56:22.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:56:23.429Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d89c3135-0565-4871-83e7-22b4e9b8ad05 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:56:23.430Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d89c3135-0565-4871-83e7-22b4e9b8ad05 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6ece82db 2021-07-13T00:56:23.430Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d89c3135-0565-4871-83e7-22b4e9b8ad05 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:56:23.430Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d89c3135-0565-4871-83e7-22b4e9b8ad05 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:56:23.860Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=d89c3135-0565-4871-83e7-22b4e9b8ad05 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:56:23.860Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=d89c3135-0565-4871-83e7-22b4e9b8ad05 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:56:24.688Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:56:25.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3d23c48-c383-4ebf-b8ae-f246eb14e8e1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:56:25.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3d23c48-c383-4ebf-b8ae-f246eb14e8e1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@92d56d4 2021-07-13T00:56:25.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3d23c48-c383-4ebf-b8ae-f246eb14e8e1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:56:25.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f3d23c48-c383-4ebf-b8ae-f246eb14e8e1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:56:26.171Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e36015f-d17a-4b1e-a4b2-c0027ea73209 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:56:26.171Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e36015f-d17a-4b1e-a4b2-c0027ea73209 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@741716df 2021-07-13T00:56:26.171Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e36015f-d17a-4b1e-a4b2-c0027ea73209 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:56:26.171Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e36015f-d17a-4b1e-a4b2-c0027ea73209 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:56:27.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31dd88a3-3af3-48d5-a841-71d843a2ee5f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:56:27.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31dd88a3-3af3-48d5-a841-71d843a2ee5f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6ed59adc 2021-07-13T00:56:27.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31dd88a3-3af3-48d5-a841-71d843a2ee5f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:56:27.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=31dd88a3-3af3-48d5-a841-71d843a2ee5f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:56:29.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73dddf64-06e5-4662-88f8-8def67e49a37 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:56:29.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73dddf64-06e5-4662-88f8-8def67e49a37 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@60a41820 2021-07-13T00:56:29.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73dddf64-06e5-4662-88f8-8def67e49a37 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:56:29.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73dddf64-06e5-4662-88f8-8def67e49a37 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:56:32.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c52e41c4-d235-4588-915b-6e2e3d05f139 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:56:32.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c52e41c4-d235-4588-915b-6e2e3d05f139 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@35c6916 2021-07-13T00:56:32.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c52e41c4-d235-4588-915b-6e2e3d05f139 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:56:32.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c52e41c4-d235-4588-915b-6e2e3d05f139 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:56:36.015Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=edd3cd36-0f48-4a32-9310-d3df36a17275 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:56:36.015Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=edd3cd36-0f48-4a32-9310-d3df36a17275 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4f31364c 2021-07-13T00:56:36.015Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=edd3cd36-0f48-4a32-9310-d3df36a17275 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:56:36.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=edd3cd36-0f48-4a32-9310-d3df36a17275 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:56:37.918Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:56:37.927Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:56:40.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:56:40.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:56:41.318Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5930fbb-2b73-42c8-a62b-b614748260c7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:56:41.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5930fbb-2b73-42c8-a62b-b614748260c7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@df0460c 2021-07-13T00:56:41.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5930fbb-2b73-42c8-a62b-b614748260c7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:56:41.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a5930fbb-2b73-42c8-a62b-b614748260c7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:56:42.760Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:56:43.720Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e4a3259-3097-4213-97d8-70070f21eb9d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:56:43.721Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e4a3259-3097-4213-97d8-70070f21eb9d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4d552a8a 2021-07-13T00:56:43.721Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e4a3259-3097-4213-97d8-70070f21eb9d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:56:43.721Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3e4a3259-3097-4213-97d8-70070f21eb9d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:56:44.359Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=3e4a3259-3097-4213-97d8-70070f21eb9d ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:56:44.359Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=3e4a3259-3097-4213-97d8-70070f21eb9d ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:56:45.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=846a221e-49de-421d-a919-10b2dbcd46e5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:56:45.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=846a221e-49de-421d-a919-10b2dbcd46e5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d21eeef 2021-07-13T00:56:45.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=846a221e-49de-421d-a919-10b2dbcd46e5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:56:45.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=846a221e-49de-421d-a919-10b2dbcd46e5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:56:45.842Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=846a221e-49de-421d-a919-10b2dbcd46e5 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:56:45.842Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=846a221e-49de-421d-a919-10b2dbcd46e5 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:56:48.222Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f98ea37-f4a4-467b-a053-9b3b9c081969 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:56:48.222Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f98ea37-f4a4-467b-a053-9b3b9c081969 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6b643e51 2021-07-13T00:56:48.222Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f98ea37-f4a4-467b-a053-9b3b9c081969 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:56:48.222Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f98ea37-f4a4-467b-a053-9b3b9c081969 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:56:51.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d6f9efe-84a5-4423-b819-3b96d03387be PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:56:51.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d6f9efe-84a5-4423-b819-3b96d03387be PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64abc517 2021-07-13T00:56:51.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d6f9efe-84a5-4423-b819-3b96d03387be PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:56:51.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2d6f9efe-84a5-4423-b819-3b96d03387be PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:56:52.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:56:52.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:56:54.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0dbfc962-4461-4ea8-872e-5f3bb9c4b9a2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:56:54.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0dbfc962-4461-4ea8-872e-5f3bb9c4b9a2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ea272 2021-07-13T00:56:54.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0dbfc962-4461-4ea8-872e-5f3bb9c4b9a2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:56:54.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0dbfc962-4461-4ea8-872e-5f3bb9c4b9a2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:56:57.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96139423-2e2f-41ae-ae95-066e3999d383 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:56:57.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96139423-2e2f-41ae-ae95-066e3999d383 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@223a6202 2021-07-13T00:56:57.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96139423-2e2f-41ae-ae95-066e3999d383 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:56:57.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=96139423-2e2f-41ae-ae95-066e3999d383 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:57:00.011Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=992249c5-be5b-4e89-aaf3-f453c7e72d9f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:57:00.011Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=992249c5-be5b-4e89-aaf3-f453c7e72d9f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@686646a2 2021-07-13T00:57:00.011Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=992249c5-be5b-4e89-aaf3-f453c7e72d9f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:57:00.011Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=992249c5-be5b-4e89-aaf3-f453c7e72d9f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:57:02.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a166bfe-11da-4833-b42b-639e8bdbbf00 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:57:02.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a166bfe-11da-4833-b42b-639e8bdbbf00 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@525fa5c4 2021-07-13T00:57:02.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a166bfe-11da-4833-b42b-639e8bdbbf00 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:57:02.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8a166bfe-11da-4833-b42b-639e8bdbbf00 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:57:05.968Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=8a166bfe-11da-4833-b42b-639e8bdbbf00 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:57:05.969Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=8a166bfe-11da-4833-b42b-639e8bdbbf00 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:57:05.969Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=8a166bfe-11da-4833-b42b-639e8bdbbf00 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:57:05.969Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=8a166bfe-11da-4833-b42b-639e8bdbbf00 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:57:07.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90d06faf-5ddf-4f0c-9454-63fc030b471d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:57:07.259Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90d06faf-5ddf-4f0c-9454-63fc030b471d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2777c0cf 2021-07-13T00:57:07.259Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90d06faf-5ddf-4f0c-9454-63fc030b471d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:57:07.259Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=90d06faf-5ddf-4f0c-9454-63fc030b471d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:57:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:57:07.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:57:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:57:09.369Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:57:09.369Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:57:09.659Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=90d06faf-5ddf-4f0c-9454-63fc030b471d ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:57:09.659Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=90d06faf-5ddf-4f0c-9454-63fc030b471d ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:57:10.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f44f4ddd-78bc-4208-bbe2-ec47c652ac8b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:57:10.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f44f4ddd-78bc-4208-bbe2-ec47c652ac8b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@c258370 2021-07-13T00:57:10.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f44f4ddd-78bc-4208-bbe2-ec47c652ac8b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:57:10.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f44f4ddd-78bc-4208-bbe2-ec47c652ac8b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:57:11.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73199426-48a3-4ca1-ae31-66991f09ddec PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:57:11.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73199426-48a3-4ca1-ae31-66991f09ddec PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@23fb76b5 2021-07-13T00:57:11.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73199426-48a3-4ca1-ae31-66991f09ddec PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:57:11.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73199426-48a3-4ca1-ae31-66991f09ddec PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:57:14.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5ebb9c8-6d3f-417c-9a42-14fc67ffe1b4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:57:14.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5ebb9c8-6d3f-417c-9a42-14fc67ffe1b4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3a13abbd 2021-07-13T00:57:14.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5ebb9c8-6d3f-417c-9a42-14fc67ffe1b4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:57:14.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5ebb9c8-6d3f-417c-9a42-14fc67ffe1b4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:57:17.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1af7b36-19c5-43a1-a6f4-b001d2d61c05 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:57:17.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1af7b36-19c5-43a1-a6f4-b001d2d61c05 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@8eff6ba 2021-07-13T00:57:17.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1af7b36-19c5-43a1-a6f4-b001d2d61c05 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:57:17.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1af7b36-19c5-43a1-a6f4-b001d2d61c05 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:57:20.972Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=545f6a16-80fd-46a0-842b-91fc461cdfde PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:57:20.972Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=545f6a16-80fd-46a0-842b-91fc461cdfde PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@28f1dad4 2021-07-13T00:57:20.972Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=545f6a16-80fd-46a0-842b-91fc461cdfde PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:57:20.972Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=545f6a16-80fd-46a0-842b-91fc461cdfde PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:57:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:57:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:57:23.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2c1bf33-ef05-434b-925c-9907c6ccd530 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:57:23.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2c1bf33-ef05-434b-925c-9907c6ccd530 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@50c75fa1 2021-07-13T00:57:23.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2c1bf33-ef05-434b-925c-9907c6ccd530 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:57:23.565Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a2c1bf33-ef05-434b-925c-9907c6ccd530 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:57:25.240Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:57:26.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c9c5564-d8fe-4649-a3d3-55efd3b57892 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:57:26.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c9c5564-d8fe-4649-a3d3-55efd3b57892 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@772e4847 2021-07-13T00:57:26.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c9c5564-d8fe-4649-a3d3-55efd3b57892 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:57:26.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0c9c5564-d8fe-4649-a3d3-55efd3b57892 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:57:31.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bdb95a0f-2a10-48c4-8ebd-25f47c4e2922 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:57:31.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bdb95a0f-2a10-48c4-8ebd-25f47c4e2922 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@833a3e 2021-07-13T00:57:31.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bdb95a0f-2a10-48c4-8ebd-25f47c4e2922 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:57:31.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bdb95a0f-2a10-48c4-8ebd-25f47c4e2922 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:57:33.169Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=997bc396-9204-417e-a6f9-e7a855f1814e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:57:33.170Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=997bc396-9204-417e-a6f9-e7a855f1814e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@26827780 2021-07-13T00:57:33.170Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=997bc396-9204-417e-a6f9-e7a855f1814e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:57:33.170Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=997bc396-9204-417e-a6f9-e7a855f1814e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:57:36.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=461c1104-b826-45f1-93e8-d019f989cf1d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:57:36.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=461c1104-b826-45f1-93e8-d019f989cf1d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4c520ea6 2021-07-13T00:57:36.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=461c1104-b826-45f1-93e8-d019f989cf1d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:57:36.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=461c1104-b826-45f1-93e8-d019f989cf1d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:57:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:57:37.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:57:39.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5dea886f-9361-4a1f-bdab-df648956ece1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:57:39.258Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5dea886f-9361-4a1f-bdab-df648956ece1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@493b1b1b 2021-07-13T00:57:39.259Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5dea886f-9361-4a1f-bdab-df648956ece1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:57:39.259Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5dea886f-9361-4a1f-bdab-df648956ece1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:57:41.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b56fc144-ed59-402d-89e4-3524d9fc2114 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:57:41.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b56fc144-ed59-402d-89e4-3524d9fc2114 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@50f08ac8 2021-07-13T00:57:41.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b56fc144-ed59-402d-89e4-3524d9fc2114 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:57:41.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b56fc144-ed59-402d-89e4-3524d9fc2114 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:57:44.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de7f6862-c6c0-4185-a16c-ed893250e4e3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:57:44.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de7f6862-c6c0-4185-a16c-ed893250e4e3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@657bd921 2021-07-13T00:57:44.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de7f6862-c6c0-4185-a16c-ed893250e4e3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:57:44.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de7f6862-c6c0-4185-a16c-ed893250e4e3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:57:48.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3cd97d07-5646-462a-9c80-635261703843 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:57:48.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3cd97d07-5646-462a-9c80-635261703843 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3980cc16 2021-07-13T00:57:48.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3cd97d07-5646-462a-9c80-635261703843 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:57:48.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3cd97d07-5646-462a-9c80-635261703843 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:57:51.494Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d0ad2ca-ef5d-4e80-8fa9-862b88479f79 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:57:51.494Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d0ad2ca-ef5d-4e80-8fa9-862b88479f79 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@28f2fbfe 2021-07-13T00:57:51.494Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d0ad2ca-ef5d-4e80-8fa9-862b88479f79 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:57:51.494Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d0ad2ca-ef5d-4e80-8fa9-862b88479f79 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:57:52.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:57:52.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:57:53.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d168a0d-41f6-4fec-8548-a651fca2b592 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:57:53.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d168a0d-41f6-4fec-8548-a651fca2b592 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7c0092e2 2021-07-13T00:57:53.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d168a0d-41f6-4fec-8548-a651fca2b592 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:57:53.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1d168a0d-41f6-4fec-8548-a651fca2b592 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:57:56.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb0c951f-80d6-421a-b719-82a079d83e27 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:57:56.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb0c951f-80d6-421a-b719-82a079d83e27 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4709fe92 2021-07-13T00:57:56.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb0c951f-80d6-421a-b719-82a079d83e27 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:57:56.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bb0c951f-80d6-421a-b719-82a079d83e27 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:57:59.880Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7341c0ed-a144-43a2-bdb1-002a033ef304 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:57:59.880Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7341c0ed-a144-43a2-bdb1-002a033ef304 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78567a19 2021-07-13T00:57:59.880Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7341c0ed-a144-43a2-bdb1-002a033ef304 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:57:59.881Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7341c0ed-a144-43a2-bdb1-002a033ef304 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:58:02.718Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25536b95-7690-4bef-a6ec-621ba45ee940 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:58:02.718Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25536b95-7690-4bef-a6ec-621ba45ee940 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@47c453e6 2021-07-13T00:58:02.718Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25536b95-7690-4bef-a6ec-621ba45ee940 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:58:02.718Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25536b95-7690-4bef-a6ec-621ba45ee940 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:58:05.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41a13d71-6f3d-469d-9c2e-f65ad694be9b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:58:05.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41a13d71-6f3d-469d-9c2e-f65ad694be9b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3f2b5e7e 2021-07-13T00:58:05.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41a13d71-6f3d-469d-9c2e-f65ad694be9b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:58:05.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=41a13d71-6f3d-469d-9c2e-f65ad694be9b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:58:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:58:07.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:58:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:58:09.369Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:58:09.369Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:58:11.560Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f35d77e3-4966-49cc-8847-557d1d678870 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:58:11.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f35d77e3-4966-49cc-8847-557d1d678870 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@375eda67 2021-07-13T00:58:11.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f35d77e3-4966-49cc-8847-557d1d678870 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:58:11.561Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f35d77e3-4966-49cc-8847-557d1d678870 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:58:13.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86d3fe13-315c-4520-a212-cdb355941487 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:58:13.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86d3fe13-315c-4520-a212-cdb355941487 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7c13e827 2021-07-13T00:58:13.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86d3fe13-315c-4520-a212-cdb355941487 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:58:13.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86d3fe13-315c-4520-a212-cdb355941487 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:58:15.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11526289-9934-4ee5-a046-30dcac8359bd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:58:15.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11526289-9934-4ee5-a046-30dcac8359bd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@19adcf20 2021-07-13T00:58:15.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11526289-9934-4ee5-a046-30dcac8359bd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:58:15.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=11526289-9934-4ee5-a046-30dcac8359bd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:58:17.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de3df2f4-8145-47ba-8a44-f2aba831e669 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:58:17.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de3df2f4-8145-47ba-8a44-f2aba831e669 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7c90bac5 2021-07-13T00:58:17.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de3df2f4-8145-47ba-8a44-f2aba831e669 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:58:17.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de3df2f4-8145-47ba-8a44-f2aba831e669 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:58:21.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=847208b7-629d-4926-bc59-02f0f870c5e2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:58:21.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=847208b7-629d-4926-bc59-02f0f870c5e2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3ddccad2 2021-07-13T00:58:21.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=847208b7-629d-4926-bc59-02f0f870c5e2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:58:21.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=847208b7-629d-4926-bc59-02f0f870c5e2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:58:22.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:58:22.925Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:58:25.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c36597e-a60f-40fa-9c2b-5e02b3d10f2d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:58:25.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c36597e-a60f-40fa-9c2b-5e02b3d10f2d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5eed4f9e 2021-07-13T00:58:25.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c36597e-a60f-40fa-9c2b-5e02b3d10f2d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:58:25.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7c36597e-a60f-40fa-9c2b-5e02b3d10f2d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:58:26.380Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:58:26.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c31290c5-50be-4ad5-945c-ed2bb52724ed PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:58:26.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c31290c5-50be-4ad5-945c-ed2bb52724ed PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@294e5fac 2021-07-13T00:58:26.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c31290c5-50be-4ad5-945c-ed2bb52724ed PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:58:26.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c31290c5-50be-4ad5-945c-ed2bb52724ed PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:58:29.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d2f6dfa-2ee7-4ccf-8878-32fc4e9c7cea PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:58:29.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d2f6dfa-2ee7-4ccf-8878-32fc4e9c7cea PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@50718ed2 2021-07-13T00:58:29.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d2f6dfa-2ee7-4ccf-8878-32fc4e9c7cea PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:58:29.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4d2f6dfa-2ee7-4ccf-8878-32fc4e9c7cea PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:58:32.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=053bb849-376a-4f40-9f12-ea173cd8b6e8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:58:32.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=053bb849-376a-4f40-9f12-ea173cd8b6e8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@b205454 2021-07-13T00:58:32.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=053bb849-376a-4f40-9f12-ea173cd8b6e8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:58:32.861Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=053bb849-376a-4f40-9f12-ea173cd8b6e8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:58:34.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:58:34.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:58:35.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00779561-b0cb-42dc-a940-a26cd167214f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:58:35.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00779561-b0cb-42dc-a940-a26cd167214f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3f856b11 2021-07-13T00:58:35.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00779561-b0cb-42dc-a940-a26cd167214f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:58:35.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00779561-b0cb-42dc-a940-a26cd167214f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:58:36.160Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=00779561-b0cb-42dc-a940-a26cd167214f ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:58:36.160Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=00779561-b0cb-42dc-a940-a26cd167214f ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:58:36.470Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:58:37.917Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:58:37.926Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:58:38.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ad49a79-9da4-437c-bf21-24209d195cd0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:58:38.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ad49a79-9da4-437c-bf21-24209d195cd0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@646df072 2021-07-13T00:58:38.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ad49a79-9da4-437c-bf21-24209d195cd0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:58:38.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7ad49a79-9da4-437c-bf21-24209d195cd0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:58:39.065Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=7ad49a79-9da4-437c-bf21-24209d195cd0 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:58:39.066Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=7ad49a79-9da4-437c-bf21-24209d195cd0 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:58:41.815Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f44bfcb-35b6-4619-a7b4-b87b377db69c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:58:41.815Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f44bfcb-35b6-4619-a7b4-b87b377db69c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@38b6f533 2021-07-13T00:58:41.815Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f44bfcb-35b6-4619-a7b4-b87b377db69c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:58:41.815Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3f44bfcb-35b6-4619-a7b4-b87b377db69c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:58:44.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=12f5650d-9762-453b-8ecc-aa08f2ecd2cb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:58:44.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=12f5650d-9762-453b-8ecc-aa08f2ecd2cb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@31bd8485 2021-07-13T00:58:44.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=12f5650d-9762-453b-8ecc-aa08f2ecd2cb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:58:44.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=12f5650d-9762-453b-8ecc-aa08f2ecd2cb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:58:47.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be227474-25bf-4684-b0b9-b58397a29be2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:58:47.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be227474-25bf-4684-b0b9-b58397a29be2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@25f4ac40 2021-07-13T00:58:47.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be227474-25bf-4684-b0b9-b58397a29be2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:58:47.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be227474-25bf-4684-b0b9-b58397a29be2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:58:50.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c15a749d-6522-4928-9281-dcd63dbce9e9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:58:50.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c15a749d-6522-4928-9281-dcd63dbce9e9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2e9126ec 2021-07-13T00:58:50.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c15a749d-6522-4928-9281-dcd63dbce9e9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:58:50.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c15a749d-6522-4928-9281-dcd63dbce9e9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:58:52.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:58:52.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:58:53.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ffc72dd-bbd3-41ee-a82f-df1b4ed535b1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:58:53.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ffc72dd-bbd3-41ee-a82f-df1b4ed535b1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@21014160 2021-07-13T00:58:53.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ffc72dd-bbd3-41ee-a82f-df1b4ed535b1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:58:53.673Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ffc72dd-bbd3-41ee-a82f-df1b4ed535b1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:58:56.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26fa4876-6810-4ea5-b587-2790c31aacd2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:58:56.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26fa4876-6810-4ea5-b587-2790c31aacd2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7cc57069 2021-07-13T00:58:56.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26fa4876-6810-4ea5-b587-2790c31aacd2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:58:56.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26fa4876-6810-4ea5-b587-2790c31aacd2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:58:59.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=457481c3-2720-4234-a35e-bd8a3ea7212c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:58:59.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=457481c3-2720-4234-a35e-bd8a3ea7212c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7cd3451d 2021-07-13T00:58:59.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=457481c3-2720-4234-a35e-bd8a3ea7212c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:58:59.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=457481c3-2720-4234-a35e-bd8a3ea7212c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:59:02.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21d49027-f1a1-4e19-81cc-c33d395cca38 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:59:02.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21d49027-f1a1-4e19-81cc-c33d395cca38 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5b52ba2d 2021-07-13T00:59:02.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21d49027-f1a1-4e19-81cc-c33d395cca38 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:59:02.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=21d49027-f1a1-4e19-81cc-c33d395cca38 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:59:05.838Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05da1a92-d507-450d-bdd3-1b1ec25720d9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:59:05.838Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05da1a92-d507-450d-bdd3-1b1ec25720d9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7073ea66 2021-07-13T00:59:05.838Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05da1a92-d507-450d-bdd3-1b1ec25720d9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:59:05.838Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05da1a92-d507-450d-bdd3-1b1ec25720d9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:59:07.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:59:07.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T00:59:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:59:07.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:59:08.844Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=05da1a92-d507-450d-bdd3-1b1ec25720d9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:59:08.845Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=05da1a92-d507-450d-bdd3-1b1ec25720d9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:59:08.845Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=05da1a92-d507-450d-bdd3-1b1ec25720d9 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:59:08.845Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=05da1a92-d507-450d-bdd3-1b1ec25720d9 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:59:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T00:59:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T00:59:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T00:59:10.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:59:13.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:59:16.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:59:19.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:59:22.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:59:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:59:22.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:59:24.525Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:59:25.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:59:28.313Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01xziz8eq5ah1y1o9f9jpjd07e0335 2021-07-13T00:59:28.314Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0mlosmc735j9817q67nqvptmus330 2021-07-13T00:59:28.314Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0180mvpwx0z19u10wgx7phhem48325 2021-07-13T00:59:28.315Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0euonodsp7wa19gpsxp7h8dwg317 2021-07-13T00:59:28.315Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0a7d4mwv70ejo10h87jh2eippd336 2021-07-13T00:59:28.315Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node08g1ws61qzfds1wtoph039tjmn333 2021-07-13T00:59:28.316Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01qw4xcn09c8vyv4ub4sjhkgss323 2021-07-13T00:59:28.316Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0oeevterzb24e1q2xc73ou6i9y326 2021-07-13T00:59:28.316Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node070ln3g7sdutx86ncu6m3uqfh329 2021-07-13T00:59:28.316Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01v9h784jfjxvu15yrouc41ukf8328 2021-07-13T00:59:28.316Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0j7k4m0lj4kxl17v36ebydsgzm332 2021-07-13T00:59:28.316Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0afpg9u0si7ngxvutd9ay0dbh327 2021-07-13T00:59:28.316Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node09svrbl7jolztecihurzmvmc0331 2021-07-13T00:59:28.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:59:28.868Z [cluster1-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.199.132:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.199.132:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T00:59:28.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8069a35-695d-45ac-bce7-f5f4092e28d4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:59:28.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8069a35-695d-45ac-bce7-f5f4092e28d4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@27628a39 2021-07-13T00:59:28.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8069a35-695d-45ac-bce7-f5f4092e28d4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:59:28.868Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b8069a35-695d-45ac-bce7-f5f4092e28d4 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:59:31.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:59:31.917Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=b8069a35-695d-45ac-bce7-f5f4092e28d4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:59:31.918Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=b8069a35-695d-45ac-bce7-f5f4092e28d4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:59:31.918Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b8069a35-695d-45ac-bce7-f5f4092e28d4 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:59:31.918Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=b8069a35-695d-45ac-bce7-f5f4092e28d4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:59:32.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0066d673-bb3a-4a84-98df-03cbad3dc31d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:59:32.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0066d673-bb3a-4a84-98df-03cbad3dc31d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@f3281a5 2021-07-13T00:59:32.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0066d673-bb3a-4a84-98df-03cbad3dc31d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:59:32.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0066d673-bb3a-4a84-98df-03cbad3dc31d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T00:59:34.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T00:59:35.418Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=0066d673-bb3a-4a84-98df-03cbad3dc31d ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:59:35.418Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=0066d673-bb3a-4a84-98df-03cbad3dc31d ErrorCategory=ERROR ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T00:59:36.168Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09fda29a-a9e2-4afc-9ebc-db8c73bba5f1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:59:36.168Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09fda29a-a9e2-4afc-9ebc-db8c73bba5f1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@53fd078b 2021-07-13T00:59:36.168Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09fda29a-a9e2-4afc-9ebc-db8c73bba5f1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:59:36.168Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=09fda29a-a9e2-4afc-9ebc-db8c73bba5f1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:59:36.662Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T00:59:37.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=443a23a7-241a-4367-aa4d-42fb288d5d9e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:59:37.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=443a23a7-241a-4367-aa4d-42fb288d5d9e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6c7e5171 2021-07-13T00:59:37.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=443a23a7-241a-4367-aa4d-42fb288d5d9e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:59:37.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=443a23a7-241a-4367-aa4d-42fb288d5d9e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:59:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:59:37.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:59:38.268Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=443a23a7-241a-4367-aa4d-42fb288d5d9e ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:59:38.268Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=443a23a7-241a-4367-aa4d-42fb288d5d9e ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:59:39.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c80fa53f-d0b0-4b92-9466-e1d7ae66b558 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:59:39.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c80fa53f-d0b0-4b92-9466-e1d7ae66b558 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@775acaf2 2021-07-13T00:59:39.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c80fa53f-d0b0-4b92-9466-e1d7ae66b558 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:59:39.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c80fa53f-d0b0-4b92-9466-e1d7ae66b558 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:59:41.022Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de90e977-d386-4629-b814-782fd2fe0aab PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:59:41.022Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de90e977-d386-4629-b814-782fd2fe0aab PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7c7b7c74 2021-07-13T00:59:41.022Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de90e977-d386-4629-b814-782fd2fe0aab PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:59:41.022Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de90e977-d386-4629-b814-782fd2fe0aab PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:59:42.473Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c3f4252-7c61-4256-b5e6-03490cec5270 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:59:42.473Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c3f4252-7c61-4256-b5e6-03490cec5270 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4701ed2f 2021-07-13T00:59:42.473Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c3f4252-7c61-4256-b5e6-03490cec5270 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:59:42.473Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8c3f4252-7c61-4256-b5e6-03490cec5270 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:59:43.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa0dd562-09a3-49e1-a316-d2182dff1a84 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:59:43.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa0dd562-09a3-49e1-a316-d2182dff1a84 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@596ad967 2021-07-13T00:59:43.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa0dd562-09a3-49e1-a316-d2182dff1a84 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:59:43.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aa0dd562-09a3-49e1-a316-d2182dff1a84 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:59:45.272Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df68a6e4-3320-43ab-a8c6-eb81f281aaba PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:59:45.272Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df68a6e4-3320-43ab-a8c6-eb81f281aaba PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78c685e2 2021-07-13T00:59:45.272Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df68a6e4-3320-43ab-a8c6-eb81f281aaba PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:59:45.272Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df68a6e4-3320-43ab-a8c6-eb81f281aaba PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:59:47.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1c42322-97f0-450d-a074-f3edae352dac PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:59:47.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1c42322-97f0-450d-a074-f3edae352dac PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5ff9de52 2021-07-13T00:59:47.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1c42322-97f0-450d-a074-f3edae352dac PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:59:47.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e1c42322-97f0-450d-a074-f3edae352dac PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:59:50.373Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=e1c42322-97f0-450d-a074-f3edae352dac ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T00:59:50.374Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=e1c42322-97f0-450d-a074-f3edae352dac ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T00:59:50.374Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=e1c42322-97f0-450d-a074-f3edae352dac ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:59:50.374Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=e1c42322-97f0-450d-a074-f3edae352dac ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:59:51.770Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b43a2221-ef92-49b8-9a92-be6f630a405a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:59:51.770Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b43a2221-ef92-49b8-9a92-be6f630a405a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@17592817 2021-07-13T00:59:51.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b43a2221-ef92-49b8-9a92-be6f630a405a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:59:51.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b43a2221-ef92-49b8-9a92-be6f630a405a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:59:52.319Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b43a2221-ef92-49b8-9a92-be6f630a405a ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:59:52.319Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b43a2221-ef92-49b8-9a92-be6f630a405a ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T00:59:52.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T00:59:52.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T00:59:53.467Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=569417be-cf90-4eea-a696-af7463519ca1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:59:53.467Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=569417be-cf90-4eea-a696-af7463519ca1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@54d21513 2021-07-13T00:59:53.467Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=569417be-cf90-4eea-a696-af7463519ca1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:59:53.467Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=569417be-cf90-4eea-a696-af7463519ca1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:59:55.215Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=19fd178c-01f1-4cdd-b39c-a9c7d2b0339a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:59:55.215Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=19fd178c-01f1-4cdd-b39c-a9c7d2b0339a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@45485a95 2021-07-13T00:59:55.215Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=19fd178c-01f1-4cdd-b39c-a9c7d2b0339a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:59:55.215Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=19fd178c-01f1-4cdd-b39c-a9c7d2b0339a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:59:56.874Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1601564c-cc48-480e-936e-ca46709e29c5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:59:56.874Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1601564c-cc48-480e-936e-ca46709e29c5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b0fcbbb 2021-07-13T00:59:56.874Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1601564c-cc48-480e-936e-ca46709e29c5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:59:56.875Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1601564c-cc48-480e-936e-ca46709e29c5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T00:59:58.818Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b9398fb-0da7-4348-9735-768c36aa2fcb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T00:59:58.818Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b9398fb-0da7-4348-9735-768c36aa2fcb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3808560a 2021-07-13T00:59:58.818Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b9398fb-0da7-4348-9735-768c36aa2fcb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T00:59:58.818Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2b9398fb-0da7-4348-9735-768c36aa2fcb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:00:00.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5b353dd-b099-4173-a3cc-49857343a97b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:00:00.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5b353dd-b099-4173-a3cc-49857343a97b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7cb6b695 2021-07-13T01:00:00.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5b353dd-b099-4173-a3cc-49857343a97b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:00:00.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5b353dd-b099-4173-a3cc-49857343a97b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:00:02.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62b6c417-b33e-4389-943b-d0e57bde399b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:00:02.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62b6c417-b33e-4389-943b-d0e57bde399b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7badd764 2021-07-13T01:00:02.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62b6c417-b33e-4389-943b-d0e57bde399b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:00:02.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62b6c417-b33e-4389-943b-d0e57bde399b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:00:05.020Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91e07b2a-b254-4d98-be13-9dd36c0ffc06 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:00:05.020Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91e07b2a-b254-4d98-be13-9dd36c0ffc06 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1e1a89b9 2021-07-13T01:00:05.020Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91e07b2a-b254-4d98-be13-9dd36c0ffc06 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:00:05.020Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=91e07b2a-b254-4d98-be13-9dd36c0ffc06 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:00:06.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=349916b6-9c77-4fc6-8697-a56ab67b0887 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:00:06.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=349916b6-9c77-4fc6-8697-a56ab67b0887 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64583476 2021-07-13T01:00:06.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=349916b6-9c77-4fc6-8697-a56ab67b0887 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:00:06.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=349916b6-9c77-4fc6-8697-a56ab67b0887 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:00:07.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70ca0226-9352-409a-b207-50e0652f4832 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:00:07.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70ca0226-9352-409a-b207-50e0652f4832 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d933476 2021-07-13T01:00:07.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70ca0226-9352-409a-b207-50e0652f4832 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:00:07.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=70ca0226-9352-409a-b207-50e0652f4832 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:00:07.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:00:07.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:00:09.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=925c9c38-5cd0-47f4-86ec-2fff77ddec70 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:00:09.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=925c9c38-5cd0-47f4-86ec-2fff77ddec70 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3810d057 2021-07-13T01:00:09.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=925c9c38-5cd0-47f4-86ec-2fff77ddec70 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:00:09.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=925c9c38-5cd0-47f4-86ec-2fff77ddec70 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:00:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T01:00:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T01:00:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T01:00:11.662Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6c3d55f-3719-464e-ba75-61901e2c1d06 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:00:11.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6c3d55f-3719-464e-ba75-61901e2c1d06 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@728dd1ad 2021-07-13T01:00:11.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6c3d55f-3719-464e-ba75-61901e2c1d06 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:00:11.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6c3d55f-3719-464e-ba75-61901e2c1d06 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:00:14.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=747858aa-0efb-4481-a7e7-103d9385ccdd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:00:14.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=747858aa-0efb-4481-a7e7-103d9385ccdd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5753a00e 2021-07-13T01:00:14.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=747858aa-0efb-4481-a7e7-103d9385ccdd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:00:14.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=747858aa-0efb-4481-a7e7-103d9385ccdd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:00:17.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dce55ffe-30a8-432f-b228-4b91285dae7e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:00:17.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dce55ffe-30a8-432f-b228-4b91285dae7e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63d71e16 2021-07-13T01:00:17.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dce55ffe-30a8-432f-b228-4b91285dae7e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:00:17.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dce55ffe-30a8-432f-b228-4b91285dae7e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:00:20.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13c9fc4c-b2bd-4cf8-8e80-c546074aa9e3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:00:20.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13c9fc4c-b2bd-4cf8-8e80-c546074aa9e3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5d7bf853 2021-07-13T01:00:20.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13c9fc4c-b2bd-4cf8-8e80-c546074aa9e3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:00:20.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13c9fc4c-b2bd-4cf8-8e80-c546074aa9e3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:00:22.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:00:22.938Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:00:24.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6822aa8d-23a4-47d6-9609-6808f1571289 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:00:24.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6822aa8d-23a4-47d6-9609-6808f1571289 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@41675ae8 2021-07-13T01:00:24.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6822aa8d-23a4-47d6-9609-6808f1571289 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:00:24.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6822aa8d-23a4-47d6-9609-6808f1571289 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:00:24.468Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:00:26.714Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61801096-2ab8-4984-8939-aced2544bcf9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:00:26.714Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61801096-2ab8-4984-8939-aced2544bcf9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@bfec11a 2021-07-13T01:00:26.714Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61801096-2ab8-4984-8939-aced2544bcf9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:00:26.714Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61801096-2ab8-4984-8939-aced2544bcf9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:00:30.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f29c2f1f-b63f-41a2-86e3-23de81dc7dec PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:00:30.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f29c2f1f-b63f-41a2-86e3-23de81dc7dec PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@508249dd 2021-07-13T01:00:30.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f29c2f1f-b63f-41a2-86e3-23de81dc7dec PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:00:30.366Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f29c2f1f-b63f-41a2-86e3-23de81dc7dec PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:00:32.871Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed9cf77d-f2c5-4fdb-8e17-715b08b11cf1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:00:32.872Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed9cf77d-f2c5-4fdb-8e17-715b08b11cf1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@52399afe 2021-07-13T01:00:32.872Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed9cf77d-f2c5-4fdb-8e17-715b08b11cf1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:00:32.872Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ed9cf77d-f2c5-4fdb-8e17-715b08b11cf1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:00:35.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e84f7223-2431-42c3-8044-1c0b75e750db PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:00:35.766Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e84f7223-2431-42c3-8044-1c0b75e750db PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@29723a4b 2021-07-13T01:00:35.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e84f7223-2431-42c3-8044-1c0b75e750db PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:00:35.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e84f7223-2431-42c3-8044-1c0b75e750db PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:00:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:00:37.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:00:38.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ae12d36-d1d4-4c1e-b582-e6ef5ab7c093 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:00:38.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ae12d36-d1d4-4c1e-b582-e6ef5ab7c093 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5479c86e 2021-07-13T01:00:38.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ae12d36-d1d4-4c1e-b582-e6ef5ab7c093 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:00:38.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3ae12d36-d1d4-4c1e-b582-e6ef5ab7c093 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:00:41.770Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ffd7668-c94d-4c33-a52c-41746ca40806 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:00:41.770Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ffd7668-c94d-4c33-a52c-41746ca40806 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@16f7af9d 2021-07-13T01:00:41.770Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ffd7668-c94d-4c33-a52c-41746ca40806 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:00:41.770Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2ffd7668-c94d-4c33-a52c-41746ca40806 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:00:44.923Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bcb971af-b260-4e0f-bb6f-9d4c96385d8f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:00:44.923Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bcb971af-b260-4e0f-bb6f-9d4c96385d8f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@88729d7 2021-07-13T01:00:44.923Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bcb971af-b260-4e0f-bb6f-9d4c96385d8f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:00:44.923Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bcb971af-b260-4e0f-bb6f-9d4c96385d8f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:00:47.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e3db49d-bc4a-4001-b3f4-63bf8b346c0d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:00:47.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e3db49d-bc4a-4001-b3f4-63bf8b346c0d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@301ffaee 2021-07-13T01:00:47.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e3db49d-bc4a-4001-b3f4-63bf8b346c0d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:00:47.963Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1e3db49d-bc4a-4001-b3f4-63bf8b346c0d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:00:49.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:00:49.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T01:00:50.967Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=1e3db49d-bc4a-4001-b3f4-63bf8b346c0d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T01:00:50.969Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=1e3db49d-bc4a-4001-b3f4-63bf8b346c0d ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T01:00:50.969Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=1e3db49d-bc4a-4001-b3f4-63bf8b346c0d ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:00:50.969Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=1e3db49d-bc4a-4001-b3f4-63bf8b346c0d ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:00:52.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:00:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:00:52.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:00:54.122Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79d28399-bdc3-4fa8-8e6d-528d01291a1c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:00:54.122Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79d28399-bdc3-4fa8-8e6d-528d01291a1c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2abf6e3b 2021-07-13T01:00:54.122Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79d28399-bdc3-4fa8-8e6d-528d01291a1c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:00:54.122Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=79d28399-bdc3-4fa8-8e6d-528d01291a1c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:00:54.666Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T01:00:57.179Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=79d28399-bdc3-4fa8-8e6d-528d01291a1c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T01:00:57.180Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=79d28399-bdc3-4fa8-8e6d-528d01291a1c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T01:00:57.180Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=79d28399-bdc3-4fa8-8e6d-528d01291a1c ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:00:57.180Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=79d28399-bdc3-4fa8-8e6d-528d01291a1c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T01:00:59.019Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7fd6622-df72-46aa-8f2a-2b90db1ba0d5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:00:59.019Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7fd6622-df72-46aa-8f2a-2b90db1ba0d5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@30e7efc3 2021-07-13T01:00:59.019Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7fd6622-df72-46aa-8f2a-2b90db1ba0d5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:00:59.019Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a7fd6622-df72-46aa-8f2a-2b90db1ba0d5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:00:59.822Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=a7fd6622-df72-46aa-8f2a-2b90db1ba0d5 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:00:59.822Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=a7fd6622-df72-46aa-8f2a-2b90db1ba0d5 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:01:00.626Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2068d56f-0844-40fa-ace2-8168e34435ca PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:01:00.626Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2068d56f-0844-40fa-ace2-8168e34435ca PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2e7fdcda 2021-07-13T01:01:00.626Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2068d56f-0844-40fa-ace2-8168e34435ca PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:01:00.626Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2068d56f-0844-40fa-ace2-8168e34435ca PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:01:02.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2554d172-254b-49e2-95b4-7f9e65f17a97 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:01:02.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2554d172-254b-49e2-95b4-7f9e65f17a97 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@52a17cb4 2021-07-13T01:01:02.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2554d172-254b-49e2-95b4-7f9e65f17a97 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:01:02.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2554d172-254b-49e2-95b4-7f9e65f17a97 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:01:04.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=393544b6-0070-41c1-b1fb-52f6f1419190 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:01:04.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=393544b6-0070-41c1-b1fb-52f6f1419190 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5d3708d4 2021-07-13T01:01:04.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=393544b6-0070-41c1-b1fb-52f6f1419190 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:01:04.458Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=393544b6-0070-41c1-b1fb-52f6f1419190 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:01:05.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d65b931-c73c-4558-b417-85f689d5e746 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:01:05.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d65b931-c73c-4558-b417-85f689d5e746 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@27705dba 2021-07-13T01:01:05.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d65b931-c73c-4558-b417-85f689d5e746 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:01:05.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0d65b931-c73c-4558-b417-85f689d5e746 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:01:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:01:07.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:01:09.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3650e56e-ec86-4bb7-8ebb-2a08a266fde2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:01:09.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3650e56e-ec86-4bb7-8ebb-2a08a266fde2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@77b9dc82 2021-07-13T01:01:09.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3650e56e-ec86-4bb7-8ebb-2a08a266fde2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:01:09.069Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3650e56e-ec86-4bb7-8ebb-2a08a266fde2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:01:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T01:01:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T01:01:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T01:01:11.922Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ffe18e76-8a44-4f2d-b34c-0a087a614445 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:01:11.922Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ffe18e76-8a44-4f2d-b34c-0a087a614445 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2e245a1d 2021-07-13T01:01:11.922Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ffe18e76-8a44-4f2d-b34c-0a087a614445 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:01:11.922Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ffe18e76-8a44-4f2d-b34c-0a087a614445 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:01:15.165Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f1d0a7d-68e0-4cf6-a5b9-afd1ebe8efa9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:01:15.166Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f1d0a7d-68e0-4cf6-a5b9-afd1ebe8efa9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4b3086a5 2021-07-13T01:01:15.166Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f1d0a7d-68e0-4cf6-a5b9-afd1ebe8efa9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:01:15.166Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6f1d0a7d-68e0-4cf6-a5b9-afd1ebe8efa9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:01:18.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=637d0a0a-63ef-4228-a69d-6c8ca79e535f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:01:18.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=637d0a0a-63ef-4228-a69d-6c8ca79e535f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6e889ff4 2021-07-13T01:01:18.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=637d0a0a-63ef-4228-a69d-6c8ca79e535f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:01:18.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=637d0a0a-63ef-4228-a69d-6c8ca79e535f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:01:21.164Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=637d0a0a-63ef-4228-a69d-6c8ca79e535f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T01:01:21.165Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=637d0a0a-63ef-4228-a69d-6c8ca79e535f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T01:01:21.165Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=637d0a0a-63ef-4228-a69d-6c8ca79e535f ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:01:21.165Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=637d0a0a-63ef-4228-a69d-6c8ca79e535f ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:01:22.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:01:22.925Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:01:25.749Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:01:31.563Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:01:31.563Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T01:01:32.563Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:01:33.564Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:01:34.564Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:01:35.564Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:01:37.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:01:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:01:37.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:01:40.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:01:41.267Z [cluster1-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.199.132:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.199.132:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T01:01:41.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c1e9507c-ec84-4640-9899-e3d63f1fc786 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:01:41.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c1e9507c-ec84-4640-9899-e3d63f1fc786 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@57ec036b 2021-07-13T01:01:41.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c1e9507c-ec84-4640-9899-e3d63f1fc786 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:01:41.268Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c1e9507c-ec84-4640-9899-e3d63f1fc786 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T01:01:43.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:01:44.311Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=c1e9507c-ec84-4640-9899-e3d63f1fc786 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T01:01:44.312Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=c1e9507c-ec84-4640-9899-e3d63f1fc786 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T01:01:45.461Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88914e9c-9986-4738-82b1-4fd279b472c1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:01:45.461Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88914e9c-9986-4738-82b1-4fd279b472c1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@14deef50 2021-07-13T01:01:45.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88914e9c-9986-4738-82b1-4fd279b472c1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:01:45.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=88914e9c-9986-4738-82b1-4fd279b472c1 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T01:01:46.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:01:48.507Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=88914e9c-9986-4738-82b1-4fd279b472c1 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T01:01:48.507Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=88914e9c-9986-4738-82b1-4fd279b472c1 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T01:01:48.507Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=88914e9c-9986-4738-82b1-4fd279b472c1 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:01:48.507Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=88914e9c-9986-4738-82b1-4fd279b472c1 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T01:01:49.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:01:49.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5d58e037-7dda-4a27-9a44-59760ac55e54 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:01:49.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5d58e037-7dda-4a27-9a44-59760ac55e54 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@9430ee0 2021-07-13T01:01:49.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5d58e037-7dda-4a27-9a44-59760ac55e54 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:01:49.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5d58e037-7dda-4a27-9a44-59760ac55e54 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T01:01:52.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:01:52.701Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=5d58e037-7dda-4a27-9a44-59760ac55e54 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T01:01:52.702Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=5d58e037-7dda-4a27-9a44-59760ac55e54 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T01:01:52.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:01:52.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:01:53.564Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e9c06ee-d571-4d08-a2a7-9e517abe3d2c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:01:53.564Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e9c06ee-d571-4d08-a2a7-9e517abe3d2c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@13103053 2021-07-13T01:01:53.564Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e9c06ee-d571-4d08-a2a7-9e517abe3d2c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:01:53.564Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4e9c06ee-d571-4d08-a2a7-9e517abe3d2c PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T01:01:55.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:01:56.605Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=4e9c06ee-d571-4d08-a2a7-9e517abe3d2c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T01:01:56.605Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=4e9c06ee-d571-4d08-a2a7-9e517abe3d2c ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T01:01:57.564Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea6024b9-2b0b-45ba-b424-925c4056bfec PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:01:57.564Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea6024b9-2b0b-45ba-b424-925c4056bfec PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@213d8079 2021-07-13T01:01:57.564Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea6024b9-2b0b-45ba-b424-925c4056bfec PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:01:57.564Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea6024b9-2b0b-45ba-b424-925c4056bfec PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T01:01:58.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:02:00.766Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T01:02:00.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e59765bd-2a8f-47ec-9c55-07d146c74956 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:02:00.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e59765bd-2a8f-47ec-9c55-07d146c74956 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7c8e9a47 2021-07-13T01:02:00.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e59765bd-2a8f-47ec-9c55-07d146c74956 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:02:00.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e59765bd-2a8f-47ec-9c55-07d146c74956 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T01:02:01.669Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc7fc993-9f73-48a4-a23b-64a16e79a3bd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:02:01.669Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc7fc993-9f73-48a4-a23b-64a16e79a3bd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@907e5de 2021-07-13T01:02:01.669Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc7fc993-9f73-48a4-a23b-64a16e79a3bd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:02:01.669Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fc7fc993-9f73-48a4-a23b-64a16e79a3bd PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T01:02:01.770Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=fc7fc993-9f73-48a4-a23b-64a16e79a3bd ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:02:01.770Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=fc7fc993-9f73-48a4-a23b-64a16e79a3bd ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:02:02.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6419aba-5d4a-4d07-a7ea-4574604f826e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:02:02.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6419aba-5d4a-4d07-a7ea-4574604f826e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@44f492cf 2021-07-13T01:02:02.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6419aba-5d4a-4d07-a7ea-4574604f826e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:02:02.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f6419aba-5d4a-4d07-a7ea-4574604f826e PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T01:02:03.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=538af30a-c8a4-448f-8bcd-5edb70b36e99 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:02:03.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=538af30a-c8a4-448f-8bcd-5edb70b36e99 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f0a3ae9 2021-07-13T01:02:03.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=538af30a-c8a4-448f-8bcd-5edb70b36e99 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:02:03.758Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=538af30a-c8a4-448f-8bcd-5edb70b36e99 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T01:02:04.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a3ca552-e923-440b-8aa4-9e36d4a1950f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:02:04.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a3ca552-e923-440b-8aa4-9e36d4a1950f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@575ecfed 2021-07-13T01:02:04.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a3ca552-e923-440b-8aa4-9e36d4a1950f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:02:04.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6a3ca552-e923-440b-8aa4-9e36d4a1950f PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T01:02:05.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=381a5394-0971-4ea1-a128-93866a17ba6d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:02:05.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=381a5394-0971-4ea1-a128-93866a17ba6d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6141f9cd 2021-07-13T01:02:05.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=381a5394-0971-4ea1-a128-93866a17ba6d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:02:05.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=381a5394-0971-4ea1-a128-93866a17ba6d PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T01:02:07.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6650e3ae-587b-4763-9ad7-1edf4ea36b82 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:02:07.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6650e3ae-587b-4763-9ad7-1edf4ea36b82 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1437dc0 2021-07-13T01:02:07.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6650e3ae-587b-4763-9ad7-1edf4ea36b82 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:02:07.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6650e3ae-587b-4763-9ad7-1edf4ea36b82 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T01:02:07.862Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6650e3ae-587b-4763-9ad7-1edf4ea36b82 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:02:07.862Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=6650e3ae-587b-4763-9ad7-1edf4ea36b82 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:02:07.917Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:02:07.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:02:08.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4986c544-bc5e-4120-826d-e83464f14489 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:02:08.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4986c544-bc5e-4120-826d-e83464f14489 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@f90d273 2021-07-13T01:02:08.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4986c544-bc5e-4120-826d-e83464f14489 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:02:08.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4986c544-bc5e-4120-826d-e83464f14489 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:02:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T01:02:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T01:02:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T01:02:11.023Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94529800-1766-49a8-81e2-f2e8fe88223b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:02:11.023Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94529800-1766-49a8-81e2-f2e8fe88223b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2e1055cb 2021-07-13T01:02:11.023Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94529800-1766-49a8-81e2-f2e8fe88223b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:02:11.023Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=94529800-1766-49a8-81e2-f2e8fe88223b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:02:16.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:02:16.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T01:02:19.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:02:22.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:02:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:02:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:02:23.225Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04f61993-7245-4340-9aaa-77a5d36d22fa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:02:23.225Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04f61993-7245-4340-9aaa-77a5d36d22fa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@40f15e55 2021-07-13T01:02:23.225Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04f61993-7245-4340-9aaa-77a5d36d22fa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:02:23.225Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=04f61993-7245-4340-9aaa-77a5d36d22fa PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:02:23.227Z [cluster2-nio-worker-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.199.132:9042 com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host cassandra.onap/10.242.199.132:9042: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadFailureException: Operation failed - received 0 responses and 1 failures at com.datastax.driver.core.Connection$9.apply(Connection.java:518) at com.datastax.driver.core.Connection$9.apply(Connection.java:480) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:213) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:202) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:118) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1509) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1262) at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1180) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T01:02:24.706Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:02:25.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:02:28.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:02:31.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:02:34.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:02:37.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:02:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:02:37.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:02:40.460Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:02:43.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:02:43.566Z [cluster1-timeouter-0] WARN c.d.driver.core.HostConnectionPool Error creating connection to cassandra.onap/10.242.199.132:9042 com.datastax.driver.core.exceptions.OperationTimedOutException: [cassandra.onap/10.242.199.132:9042] Operation timed out at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T01:02:43.567Z [cluster1-timeouter-0] WARN com.datastax.driver.core.Session Error creating pool to cassandra.onap/10.242.199.132:9042 com.datastax.driver.core.exceptions.ConnectionException: [cassandra.onap/10.242.199.132:9042] Pool was closed during initialization at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:173) at com.datastax.driver.core.HostConnectionPool$2.onSuccess(HostConnectionPool.java:158) at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1080) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:746) at com.google.common.util.concurrent.CollectionFuture.handleAllCompleted(CollectionFuture.java:69) at com.google.common.util.concurrent.AggregateFuture.processCompleted(AggregateFuture.java:282) at com.google.common.util.concurrent.AggregateFuture.decrementCountAndMaybeComplete(AggregateFuture.java:264) at com.google.common.util.concurrent.AggregateFuture.access$200(AggregateFuture.java:42) at com.google.common.util.concurrent.AggregateFuture$1.run(AggregateFuture.java:147) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:100) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771) at com.datastax.driver.core.Connection$Future.onTimeout(Connection.java:1526) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) 2021-07-13T01:02:43.567Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f2ff96b9-e427-4af1-9e5c-c2decc5ffba9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:02:43.567Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f2ff96b9-e427-4af1-9e5c-c2decc5ffba9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5a4696f6 2021-07-13T01:02:43.567Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f2ff96b9-e427-4af1-9e5c-c2decc5ffba9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:02:43.567Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f2ff96b9-e427-4af1-9e5c-c2decc5ffba9 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T01:02:46.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:02:46.588Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=f2ff96b9-e427-4af1-9e5c-c2decc5ffba9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out 2021-07-13T01:02:46.589Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=f2ff96b9-e427-4af1-9e5c-c2decc5ffba9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T01:02:46.589Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f2ff96b9-e427-4af1-9e5c-c2decc5ffba9 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:02:46.589Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=f2ff96b9-e427-4af1-9e5c-c2decc5ffba9 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T01:02:47.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c63cde37-85ac-4c43-93fc-1f511e915ae5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:02:47.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c63cde37-85ac-4c43-93fc-1f511e915ae5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7f5e7d38 2021-07-13T01:02:47.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c63cde37-85ac-4c43-93fc-1f511e915ae5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:02:47.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c63cde37-85ac-4c43-93fc-1f511e915ae5 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T01:02:49.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:02:50.768Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=c63cde37-85ac-4c43-93fc-1f511e915ae5 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out 2021-07-13T01:02:50.768Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=c63cde37-85ac-4c43-93fc-1f511e915ae5 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T01:02:51.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66288730-8ca8-4c58-8436-67d384e48333 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:02:51.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66288730-8ca8-4c58-8436-67d384e48333 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5b085602 2021-07-13T01:02:51.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66288730-8ca8-4c58-8436-67d384e48333 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:02:51.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=66288730-8ca8-4c58-8436-67d384e48333 PartnerName=SDC.BE The cassandra down nodes number is 1 2021-07-13T01:02:52.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:02:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:02:52.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:02:54.522Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T01:02:54.872Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=66288730-8ca8-4c58-8436-67d384e48333 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out 2021-07-13T01:02:54.872Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=66288730-8ca8-4c58-8436-67d384e48333 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T01:02:56.169Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25b40985-0327-41bb-9472-eb4b1d900ce4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:02:56.169Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25b40985-0327-41bb-9472-eb4b1d900ce4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@62ee4e47 2021-07-13T01:02:56.169Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25b40985-0327-41bb-9472-eb4b1d900ce4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:02:56.169Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=25b40985-0327-41bb-9472-eb4b1d900ce4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:02:59.184Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=25b40985-0327-41bb-9472-eb4b1d900ce4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out 2021-07-13T01:02:59.185Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=25b40985-0327-41bb-9472-eb4b1d900ce4 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to sdc-onboarding-be.onap:8445 [sdc-onboarding-be.onap/10.96.196.71] failed: Read timed out at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 22 common frames omitted 2021-07-13T01:02:59.185Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=25b40985-0327-41bb-9472-eb4b1d900ce4 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:02:59.185Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=25b40985-0327-41bb-9472-eb4b1d900ce4 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:03:00.360Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f072f52d-98ff-4155-9c32-e8728514ba66 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:03:00.360Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f072f52d-98ff-4155-9c32-e8728514ba66 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@726a8a40 2021-07-13T01:03:00.360Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f072f52d-98ff-4155-9c32-e8728514ba66 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:03:00.360Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f072f52d-98ff-4155-9c32-e8728514ba66 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:03:04.990Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=f072f52d-98ff-4155-9c32-e8728514ba66 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T01:03:04.990Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=f072f52d-98ff-4155-9c32-e8728514ba66 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T01:03:07.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:03:07.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T01:03:07.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:03:07.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:03:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T01:03:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T01:03:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T01:03:09.471Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T01:03:10.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f70a612-8ffc-4fbb-bfff-48b9be6d4b03 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:03:10.017Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f70a612-8ffc-4fbb-bfff-48b9be6d4b03 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@26bfec7b 2021-07-13T01:03:10.017Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f70a612-8ffc-4fbb-bfff-48b9be6d4b03 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:03:10.017Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2f70a612-8ffc-4fbb-bfff-48b9be6d4b03 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:03:13.101Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=2f70a612-8ffc-4fbb-bfff-48b9be6d4b03 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T01:03:13.102Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=2f70a612-8ffc-4fbb-bfff-48b9be6d4b03 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T01:03:15.167Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b62f7fc3-3893-48c6-ae57-be6edd981a33 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:03:15.167Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b62f7fc3-3893-48c6-ae57-be6edd981a33 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@301149cb 2021-07-13T01:03:15.167Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b62f7fc3-3893-48c6-ae57-be6edd981a33 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:03:15.167Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b62f7fc3-3893-48c6-ae57-be6edd981a33 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:03:15.667Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b62f7fc3-3893-48c6-ae57-be6edd981a33 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:03:15.667Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b62f7fc3-3893-48c6-ae57-be6edd981a33 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:03:16.658Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de9734b6-f9d9-415d-9a9a-66030045fe33 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:03:16.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de9734b6-f9d9-415d-9a9a-66030045fe33 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@478de3bb 2021-07-13T01:03:16.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de9734b6-f9d9-415d-9a9a-66030045fe33 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:03:16.659Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de9734b6-f9d9-415d-9a9a-66030045fe33 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:03:19.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8dd60646-0ec6-4aa1-af75-79dadff6f8f1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:03:19.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8dd60646-0ec6-4aa1-af75-79dadff6f8f1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2e255363 2021-07-13T01:03:19.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8dd60646-0ec6-4aa1-af75-79dadff6f8f1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:03:19.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8dd60646-0ec6-4aa1-af75-79dadff6f8f1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:03:21.620Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f2bfeb0f-4bd3-434b-9290-9284f5a6bc4a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:03:21.620Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f2bfeb0f-4bd3-434b-9290-9284f5a6bc4a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1440187d 2021-07-13T01:03:21.620Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f2bfeb0f-4bd3-434b-9290-9284f5a6bc4a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:03:21.620Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f2bfeb0f-4bd3-434b-9290-9284f5a6bc4a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:03:22.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:03:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:03:24.369Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:03:24.715Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e78f6015-08cc-4f7c-8b35-10e8226303a8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:03:24.715Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e78f6015-08cc-4f7c-8b35-10e8226303a8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4ad7f3f0 2021-07-13T01:03:24.715Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e78f6015-08cc-4f7c-8b35-10e8226303a8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:03:24.715Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e78f6015-08cc-4f7c-8b35-10e8226303a8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:03:27.618Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7f7661a-987e-45ea-a3c1-8663d4939926 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:03:27.618Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7f7661a-987e-45ea-a3c1-8663d4939926 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@62ef512a 2021-07-13T01:03:27.618Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7f7661a-987e-45ea-a3c1-8663d4939926 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:03:27.618Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7f7661a-987e-45ea-a3c1-8663d4939926 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:03:30.621Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=e7f7661a-987e-45ea-a3c1-8663d4939926 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T01:03:30.621Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=e7f7661a-987e-45ea-a3c1-8663d4939926 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T01:03:30.621Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=e7f7661a-987e-45ea-a3c1-8663d4939926 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:03:30.621Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=e7f7661a-987e-45ea-a3c1-8663d4939926 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:03:32.118Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=99a636ac-767d-46f1-83bd-443d949d9bff PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:03:32.118Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=99a636ac-767d-46f1-83bd-443d949d9bff PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6655b85f 2021-07-13T01:03:32.118Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=99a636ac-767d-46f1-83bd-443d949d9bff PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:03:32.118Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=99a636ac-767d-46f1-83bd-443d949d9bff PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:03:32.764Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=99a636ac-767d-46f1-83bd-443d949d9bff ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:03:32.764Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=99a636ac-767d-46f1-83bd-443d949d9bff ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:03:33.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e20e0e0e-e893-43de-9e5e-06875cc5aba9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:03:33.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e20e0e0e-e893-43de-9e5e-06875cc5aba9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@77ff2c3 2021-07-13T01:03:33.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e20e0e0e-e893-43de-9e5e-06875cc5aba9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:03:33.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e20e0e0e-e893-43de-9e5e-06875cc5aba9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:03:35.029Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51008d88-aec9-4271-82e9-65cac656ad0f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:03:35.030Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51008d88-aec9-4271-82e9-65cac656ad0f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f819507 2021-07-13T01:03:35.030Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51008d88-aec9-4271-82e9-65cac656ad0f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:03:35.030Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51008d88-aec9-4271-82e9-65cac656ad0f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:03:36.846Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2707ab81-7a8d-49ae-a2a8-59d68b733dc3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:03:36.846Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2707ab81-7a8d-49ae-a2a8-59d68b733dc3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@545d51e8 2021-07-13T01:03:36.846Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2707ab81-7a8d-49ae-a2a8-59d68b733dc3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:03:36.846Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2707ab81-7a8d-49ae-a2a8-59d68b733dc3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:03:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:03:37.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:03:39.938Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3394785a-1371-415a-8790-50d44930b9fb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:03:39.938Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3394785a-1371-415a-8790-50d44930b9fb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@30c3266c 2021-07-13T01:03:39.938Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3394785a-1371-415a-8790-50d44930b9fb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:03:39.938Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3394785a-1371-415a-8790-50d44930b9fb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:03:43.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49566fb2-aa2e-4623-97ae-21cc373617c2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:03:43.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49566fb2-aa2e-4623-97ae-21cc373617c2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@48d2235d 2021-07-13T01:03:43.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49566fb2-aa2e-4623-97ae-21cc373617c2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:03:43.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=49566fb2-aa2e-4623-97ae-21cc373617c2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:03:44.266Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5ea074b-35c4-4fe0-bc38-b9c1cfc687db PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:03:44.266Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5ea074b-35c4-4fe0-bc38-b9c1cfc687db PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5b68beed 2021-07-13T01:03:44.266Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5ea074b-35c4-4fe0-bc38-b9c1cfc687db PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:03:44.267Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5ea074b-35c4-4fe0-bc38-b9c1cfc687db PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:03:45.262Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42deab76-7158-4bfd-ae8c-3841eb178df0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:03:45.263Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42deab76-7158-4bfd-ae8c-3841eb178df0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@65847679 2021-07-13T01:03:45.263Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42deab76-7158-4bfd-ae8c-3841eb178df0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:03:45.263Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=42deab76-7158-4bfd-ae8c-3841eb178df0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:03:46.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3e9e2b5-13b4-4c0f-9ca6-b01070a99b1d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:03:46.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3e9e2b5-13b4-4c0f-9ca6-b01070a99b1d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4c5dcce3 2021-07-13T01:03:46.756Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3e9e2b5-13b4-4c0f-9ca6-b01070a99b1d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:03:46.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a3e9e2b5-13b4-4c0f-9ca6-b01070a99b1d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:03:48.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5f5a255-bbda-4b4e-8d9b-291f0a93f60b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:03:48.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5f5a255-bbda-4b4e-8d9b-291f0a93f60b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@15ccc6fe 2021-07-13T01:03:48.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5f5a255-bbda-4b4e-8d9b-291f0a93f60b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:03:48.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5f5a255-bbda-4b4e-8d9b-291f0a93f60b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:03:51.070Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=f5f5a255-bbda-4b4e-8d9b-291f0a93f60b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T01:03:51.072Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=f5f5a255-bbda-4b4e-8d9b-291f0a93f60b ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T01:03:51.072Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f5f5a255-bbda-4b4e-8d9b-291f0a93f60b ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:03:51.073Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=f5f5a255-bbda-4b4e-8d9b-291f0a93f60b ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:03:52.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:03:52.929Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:03:57.173Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3dbccad7-fe15-4623-87b7-ce79a05ae70b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:03:57.174Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3dbccad7-fe15-4623-87b7-ce79a05ae70b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@b4d8d7c 2021-07-13T01:03:57.174Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3dbccad7-fe15-4623-87b7-ce79a05ae70b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:03:57.174Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3dbccad7-fe15-4623-87b7-ce79a05ae70b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:03:59.763Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=3dbccad7-fe15-4623-87b7-ce79a05ae70b ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:03:59.763Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=3dbccad7-fe15-4623-87b7-ce79a05ae70b ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:04:00.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de88bd04-3b4a-4f5e-90ab-bba79aea760d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:00.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de88bd04-3b4a-4f5e-90ab-bba79aea760d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6fa4e242 2021-07-13T01:04:00.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de88bd04-3b4a-4f5e-90ab-bba79aea760d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:00.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=de88bd04-3b4a-4f5e-90ab-bba79aea760d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:02.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da8bbd59-7929-44f6-997c-211600c1eb61 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:02.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da8bbd59-7929-44f6-997c-211600c1eb61 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c2c327e 2021-07-13T01:04:02.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da8bbd59-7929-44f6-997c-211600c1eb61 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:02.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da8bbd59-7929-44f6-997c-211600c1eb61 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:03.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35a6cb2a-c7fc-4e9d-8afd-dec7fd24c573 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:03.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35a6cb2a-c7fc-4e9d-8afd-dec7fd24c573 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7b6d6aea 2021-07-13T01:04:03.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35a6cb2a-c7fc-4e9d-8afd-dec7fd24c573 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:03.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=35a6cb2a-c7fc-4e9d-8afd-dec7fd24c573 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:05.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b188a17a-8513-4a7a-bef4-3a6312967cc2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:05.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b188a17a-8513-4a7a-bef4-3a6312967cc2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@56d2f2c0 2021-07-13T01:04:05.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b188a17a-8513-4a7a-bef4-3a6312967cc2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:05.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b188a17a-8513-4a7a-bef4-3a6312967cc2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:07.917Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:04:07.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:04:08.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=415a2fa9-77e9-4c68-9433-623178456df1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:08.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=415a2fa9-77e9-4c68-9433-623178456df1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1eacbcaa 2021-07-13T01:04:08.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=415a2fa9-77e9-4c68-9433-623178456df1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:08.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=415a2fa9-77e9-4c68-9433-623178456df1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T01:04:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T01:04:09.369Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T01:04:09.626Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cce0cb5f-3c7b-4683-843f-9b2ee93da00c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:09.626Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cce0cb5f-3c7b-4683-843f-9b2ee93da00c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63eed2c3 2021-07-13T01:04:09.626Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cce0cb5f-3c7b-4683-843f-9b2ee93da00c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:09.626Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cce0cb5f-3c7b-4683-843f-9b2ee93da00c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:13.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:04:13.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T01:04:15.562Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T01:04:15.620Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=783a5e5a-dede-4a77-89e2-ebbd36ad2766 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:15.620Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=783a5e5a-dede-4a77-89e2-ebbd36ad2766 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7344de6e 2021-07-13T01:04:15.621Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=783a5e5a-dede-4a77-89e2-ebbd36ad2766 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:15.621Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=783a5e5a-dede-4a77-89e2-ebbd36ad2766 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:17.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ffcd3408-c8f9-4fdf-b735-f109dd173005 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:17.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ffcd3408-c8f9-4fdf-b735-f109dd173005 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@32c17634 2021-07-13T01:04:17.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ffcd3408-c8f9-4fdf-b735-f109dd173005 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:17.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ffcd3408-c8f9-4fdf-b735-f109dd173005 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:04:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:04:22.926Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7d7f8950-2da7-4200-9e81-66ecffefd307 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:22.926Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7d7f8950-2da7-4200-9e81-66ecffefd307 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4973d52a 2021-07-13T01:04:22.926Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7d7f8950-2da7-4200-9e81-66ecffefd307 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:22.926Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7d7f8950-2da7-4200-9e81-66ecffefd307 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:24.395Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:04:24.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3808c40-cb28-4298-a002-b1a028fe7ba4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:24.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3808c40-cb28-4298-a002-b1a028fe7ba4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5c1e7e0d 2021-07-13T01:04:24.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3808c40-cb28-4298-a002-b1a028fe7ba4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:24.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3808c40-cb28-4298-a002-b1a028fe7ba4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:28.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05720a67-bd47-4f0b-89da-5fb362419d96 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:28.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05720a67-bd47-4f0b-89da-5fb362419d96 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@97c7c4e 2021-07-13T01:04:28.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05720a67-bd47-4f0b-89da-5fb362419d96 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:28.313Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=05720a67-bd47-4f0b-89da-5fb362419d96 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:32.918Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be398c71-491d-4a78-9033-ade2c5779820 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:32.918Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be398c71-491d-4a78-9033-ade2c5779820 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3035db83 2021-07-13T01:04:32.918Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be398c71-491d-4a78-9033-ade2c5779820 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:32.918Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=be398c71-491d-4a78-9033-ade2c5779820 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:35.777Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e8531f8-cd47-4c95-a1b2-75d63c7fa254 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:35.777Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e8531f8-cd47-4c95-a1b2-75d63c7fa254 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@67b8822a 2021-07-13T01:04:35.777Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e8531f8-cd47-4c95-a1b2-75d63c7fa254 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:35.777Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e8531f8-cd47-4c95-a1b2-75d63c7fa254 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:37.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:04:37.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T01:04:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:04:37.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:04:39.218Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=432edd68-c9cd-4de5-a97f-f8818e428d8d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:39.218Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=432edd68-c9cd-4de5-a97f-f8818e428d8d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@487976c6 2021-07-13T01:04:39.218Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=432edd68-c9cd-4de5-a97f-f8818e428d8d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:39.218Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=432edd68-c9cd-4de5-a97f-f8818e428d8d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:39.659Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T01:04:40.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64b20604-7ad2-4467-9b10-40e7a25792a6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:40.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64b20604-7ad2-4467-9b10-40e7a25792a6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@220ded8b 2021-07-13T01:04:40.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64b20604-7ad2-4467-9b10-40e7a25792a6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:40.359Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=64b20604-7ad2-4467-9b10-40e7a25792a6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:40.862Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=64b20604-7ad2-4467-9b10-40e7a25792a6 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:04:40.862Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=64b20604-7ad2-4467-9b10-40e7a25792a6 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T01:04:42.582Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7b75c83-efe2-4ecf-b603-f57e26d1a228 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:42.582Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7b75c83-efe2-4ecf-b603-f57e26d1a228 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3c7ad38a 2021-07-13T01:04:42.582Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7b75c83-efe2-4ecf-b603-f57e26d1a228 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:42.582Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d7b75c83-efe2-4ecf-b603-f57e26d1a228 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:42.877Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=d7b75c83-efe2-4ecf-b603-f57e26d1a228 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:04:42.877Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=d7b75c83-efe2-4ecf-b603-f57e26d1a228 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:04:43.878Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e72e885-b5ea-4c7d-943a-de60f8ecc808 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:43.878Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e72e885-b5ea-4c7d-943a-de60f8ecc808 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f70affd 2021-07-13T01:04:43.878Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e72e885-b5ea-4c7d-943a-de60f8ecc808 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:43.878Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6e72e885-b5ea-4c7d-943a-de60f8ecc808 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:45.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=555b730a-e7e9-486b-8f49-4312e71dcff6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:45.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=555b730a-e7e9-486b-8f49-4312e71dcff6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c3c3593 2021-07-13T01:04:45.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=555b730a-e7e9-486b-8f49-4312e71dcff6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:45.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=555b730a-e7e9-486b-8f49-4312e71dcff6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:46.556Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3b46ed6-7c6a-4da7-96aa-86ba4b355386 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:46.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3b46ed6-7c6a-4da7-96aa-86ba4b355386 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5190e22a 2021-07-13T01:04:46.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3b46ed6-7c6a-4da7-96aa-86ba4b355386 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:46.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3b46ed6-7c6a-4da7-96aa-86ba4b355386 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:48.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51237090-7b9e-481f-9a1e-20020facc3aa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:48.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51237090-7b9e-481f-9a1e-20020facc3aa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3b9c9d11 2021-07-13T01:04:48.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51237090-7b9e-481f-9a1e-20020facc3aa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:48.068Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51237090-7b9e-481f-9a1e-20020facc3aa PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:49.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6056bb7-7a3f-43e7-96f3-34b7e7fd3539 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:49.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6056bb7-7a3f-43e7-96f3-34b7e7fd3539 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2189d2d8 2021-07-13T01:04:49.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6056bb7-7a3f-43e7-96f3-34b7e7fd3539 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:49.574Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b6056bb7-7a3f-43e7-96f3-34b7e7fd3539 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:51.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=065a7fd2-3f5f-4cd2-aa09-03fe6657a1be PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:51.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=065a7fd2-3f5f-4cd2-aa09-03fe6657a1be PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5ce4ebd7 2021-07-13T01:04:51.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=065a7fd2-3f5f-4cd2-aa09-03fe6657a1be PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:51.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=065a7fd2-3f5f-4cd2-aa09-03fe6657a1be PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:52.355Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00ad61ab-d970-45b4-83c2-c0b5f484cb36 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:52.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00ad61ab-d970-45b4-83c2-c0b5f484cb36 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@a33c3d3 2021-07-13T01:04:52.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00ad61ab-d970-45b4-83c2-c0b5f484cb36 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:52.356Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=00ad61ab-d970-45b4-83c2-c0b5f484cb36 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:52.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:04:52.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:04:53.564Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0010da03-678d-4a1c-a0ed-d79a56362abb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:53.564Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0010da03-678d-4a1c-a0ed-d79a56362abb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5598b391 2021-07-13T01:04:53.564Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0010da03-678d-4a1c-a0ed-d79a56362abb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:53.564Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0010da03-678d-4a1c-a0ed-d79a56362abb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:55.362Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03cce2f2-210a-4919-b1ff-c27209057d1d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:55.362Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03cce2f2-210a-4919-b1ff-c27209057d1d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3ae8b4e8 2021-07-13T01:04:55.362Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03cce2f2-210a-4919-b1ff-c27209057d1d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:55.362Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=03cce2f2-210a-4919-b1ff-c27209057d1d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:57.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae678598-49b4-49f1-835f-3666c6bb7d84 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:57.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae678598-49b4-49f1-835f-3666c6bb7d84 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@41a95a31 2021-07-13T01:04:57.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae678598-49b4-49f1-835f-3666c6bb7d84 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:57.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ae678598-49b4-49f1-835f-3666c6bb7d84 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:04:58.572Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=988f4e1e-5b4d-456d-8590-a9a2ab384967 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:04:58.573Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=988f4e1e-5b4d-456d-8590-a9a2ab384967 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5590348c 2021-07-13T01:04:58.573Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=988f4e1e-5b4d-456d-8590-a9a2ab384967 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:04:58.573Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=988f4e1e-5b4d-456d-8590-a9a2ab384967 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:05:00.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2bb7837-82b8-41e6-bdd6-007d488f89c9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:05:00.669Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2bb7837-82b8-41e6-bdd6-007d488f89c9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@45041194 2021-07-13T01:05:00.669Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2bb7837-82b8-41e6-bdd6-007d488f89c9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:05:00.669Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b2bb7837-82b8-41e6-bdd6-007d488f89c9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:05:02.556Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc3156d7-5c73-454d-a695-516372c3f997 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:05:02.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc3156d7-5c73-454d-a695-516372c3f997 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3a34a228 2021-07-13T01:05:02.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc3156d7-5c73-454d-a695-516372c3f997 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:05:02.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc3156d7-5c73-454d-a695-516372c3f997 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:05:03.917Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1dcaeae-c627-451b-8663-760233e8e235 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:05:03.917Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1dcaeae-c627-451b-8663-760233e8e235 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@31c70f8d 2021-07-13T01:05:03.917Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1dcaeae-c627-451b-8663-760233e8e235 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:05:03.917Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f1dcaeae-c627-451b-8663-760233e8e235 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:05:05.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc310aca-215b-4f3a-a493-1b140479cd5d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:05:05.567Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc310aca-215b-4f3a-a493-1b140479cd5d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@65a7796d 2021-07-13T01:05:05.567Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc310aca-215b-4f3a-a493-1b140479cd5d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:05:05.567Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dc310aca-215b-4f3a-a493-1b140479cd5d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:05:07.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:05:07.925Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:05:08.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=328856d4-cf1b-4c81-b39b-80d80443b227 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:05:08.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=328856d4-cf1b-4c81-b39b-80d80443b227 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4a5cfc 2021-07-13T01:05:08.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=328856d4-cf1b-4c81-b39b-80d80443b227 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:05:08.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=328856d4-cf1b-4c81-b39b-80d80443b227 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:05:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T01:05:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T01:05:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T01:05:09.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a48191c9-c27b-442c-bdf5-aae427cb0434 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:05:09.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a48191c9-c27b-442c-bdf5-aae427cb0434 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1ff08b3f 2021-07-13T01:05:09.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a48191c9-c27b-442c-bdf5-aae427cb0434 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:05:09.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a48191c9-c27b-442c-bdf5-aae427cb0434 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:05:13.513Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df715d33-04a9-480d-af9d-d75f2aea0392 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:05:13.513Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df715d33-04a9-480d-af9d-d75f2aea0392 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@16c2542b 2021-07-13T01:05:13.513Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df715d33-04a9-480d-af9d-d75f2aea0392 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:05:13.513Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df715d33-04a9-480d-af9d-d75f2aea0392 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:05:15.260Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd907a16-08cb-415e-b444-faeff679187b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:05:15.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd907a16-08cb-415e-b444-faeff679187b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@48703b8f 2021-07-13T01:05:15.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd907a16-08cb-415e-b444-faeff679187b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:05:15.261Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dd907a16-08cb-415e-b444-faeff679187b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:05:16.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=57166914-9fb7-47c6-9325-b6f5393fbab1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:05:16.462Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=57166914-9fb7-47c6-9325-b6f5393fbab1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1a0f23c1 2021-07-13T01:05:16.463Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=57166914-9fb7-47c6-9325-b6f5393fbab1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:05:16.463Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=57166914-9fb7-47c6-9325-b6f5393fbab1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:05:18.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6783b9a1-37e4-43ec-8e82-362562b935d9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:05:18.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6783b9a1-37e4-43ec-8e82-362562b935d9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@13deacc 2021-07-13T01:05:18.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6783b9a1-37e4-43ec-8e82-362562b935d9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:05:18.559Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=6783b9a1-37e4-43ec-8e82-362562b935d9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:05:20.660Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=82b8afe5-7ca6-4267-b706-e6b2508737ab PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:05:20.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=82b8afe5-7ca6-4267-b706-e6b2508737ab PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5da85cc 2021-07-13T01:05:20.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=82b8afe5-7ca6-4267-b706-e6b2508737ab PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:05:20.661Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=82b8afe5-7ca6-4267-b706-e6b2508737ab PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:05:22.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=edec9980-f834-4a8e-934b-5923aba10d82 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:05:22.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=edec9980-f834-4a8e-934b-5923aba10d82 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4cbf7abb 2021-07-13T01:05:22.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=edec9980-f834-4a8e-934b-5923aba10d82 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:05:22.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=edec9980-f834-4a8e-934b-5923aba10d82 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:05:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:05:22.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:05:24.581Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:05:24.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3a81b33-1468-4e66-8328-9292b2f734c8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:05:24.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3a81b33-1468-4e66-8328-9292b2f734c8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2f4d49c6 2021-07-13T01:05:24.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3a81b33-1468-4e66-8328-9292b2f734c8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:05:24.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e3a81b33-1468-4e66-8328-9292b2f734c8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:05:26.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fefa2adc-7c61-4e56-aaca-c5a0a0d49acc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:05:26.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fefa2adc-7c61-4e56-aaca-c5a0a0d49acc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@41c8bd69 2021-07-13T01:05:26.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fefa2adc-7c61-4e56-aaca-c5a0a0d49acc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:05:26.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fefa2adc-7c61-4e56-aaca-c5a0a0d49acc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:05:28.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b94ddf60-37d3-421c-ad69-dd88c3534da6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:05:28.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b94ddf60-37d3-421c-ad69-dd88c3534da6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@106da479 2021-07-13T01:05:28.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b94ddf60-37d3-421c-ad69-dd88c3534da6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:05:28.459Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b94ddf60-37d3-421c-ad69-dd88c3534da6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:05:30.630Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08cd7017-9828-4089-91b8-6e9fba596e82 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:05:30.630Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08cd7017-9828-4089-91b8-6e9fba596e82 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@27b06e6d 2021-07-13T01:05:30.630Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08cd7017-9828-4089-91b8-6e9fba596e82 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:05:30.630Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=08cd7017-9828-4089-91b8-6e9fba596e82 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:05:33.637Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=08cd7017-9828-4089-91b8-6e9fba596e82 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T01:05:33.639Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=08cd7017-9828-4089-91b8-6e9fba596e82 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T01:05:33.639Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=08cd7017-9828-4089-91b8-6e9fba596e82 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:05:33.639Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=08cd7017-9828-4089-91b8-6e9fba596e82 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:05:35.030Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9719ce4c-b76f-4d51-b37b-5bae771682ab PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:05:35.030Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9719ce4c-b76f-4d51-b37b-5bae771682ab PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@62760024 2021-07-13T01:05:35.030Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9719ce4c-b76f-4d51-b37b-5bae771682ab PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:05:35.030Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9719ce4c-b76f-4d51-b37b-5bae771682ab PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:05:37.263Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9719ce4c-b76f-4d51-b37b-5bae771682ab ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:05:37.264Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=9719ce4c-b76f-4d51-b37b-5bae771682ab ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:05:37.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:05:37.928Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:05:41.135Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73a7520c-5684-41f2-9d36-44b59b4f506d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:05:41.135Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73a7520c-5684-41f2-9d36-44b59b4f506d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b62f429 2021-07-13T01:05:41.135Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73a7520c-5684-41f2-9d36-44b59b4f506d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:05:41.136Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=73a7520c-5684-41f2-9d36-44b59b4f506d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:05:46.330Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=57f6b75b-d708-44ee-94ef-d82a71871c79 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:05:46.330Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=57f6b75b-d708-44ee-94ef-d82a71871c79 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3f85322e 2021-07-13T01:05:46.330Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=57f6b75b-d708-44ee-94ef-d82a71871c79 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:05:46.331Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=57f6b75b-d708-44ee-94ef-d82a71871c79 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:05:51.336Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2eb2203c-6cc2-413a-b944-53f6fdf94f87 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:05:51.336Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2eb2203c-6cc2-413a-b944-53f6fdf94f87 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6d2062da 2021-07-13T01:05:51.336Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2eb2203c-6cc2-413a-b944-53f6fdf94f87 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:05:51.336Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2eb2203c-6cc2-413a-b944-53f6fdf94f87 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:05:52.916Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:05:52.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:05:54.023Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a17e5b33-17ca-424e-ad4e-cee8b1816641 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:05:54.023Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a17e5b33-17ca-424e-ad4e-cee8b1816641 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@54e93e2 2021-07-13T01:05:54.023Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a17e5b33-17ca-424e-ad4e-cee8b1816641 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:05:54.023Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a17e5b33-17ca-424e-ad4e-cee8b1816641 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:05:56.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=206caaf3-bd89-4fde-993f-3beaea02e4ff PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:05:56.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=206caaf3-bd89-4fde-993f-3beaea02e4ff PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@981ecde 2021-07-13T01:05:56.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=206caaf3-bd89-4fde-993f-3beaea02e4ff PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:05:56.417Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=206caaf3-bd89-4fde-993f-3beaea02e4ff PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:05:58.524Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=028ccfd1-46bd-4f3d-8183-69c377036f12 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:05:58.524Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=028ccfd1-46bd-4f3d-8183-69c377036f12 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1f83bb82 2021-07-13T01:05:58.524Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=028ccfd1-46bd-4f3d-8183-69c377036f12 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:05:58.524Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=028ccfd1-46bd-4f3d-8183-69c377036f12 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:00.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=851778d9-0d86-40c0-a8b1-940de84a6522 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:00.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=851778d9-0d86-40c0-a8b1-940de84a6522 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@715b3c1a 2021-07-13T01:06:00.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=851778d9-0d86-40c0-a8b1-940de84a6522 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:00.319Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=851778d9-0d86-40c0-a8b1-940de84a6522 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:01.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=67a40e0c-db2d-4baf-8315-05c651a37dd4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:01.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=67a40e0c-db2d-4baf-8315-05c651a37dd4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@fb7050 2021-07-13T01:06:01.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=67a40e0c-db2d-4baf-8315-05c651a37dd4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:01.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=67a40e0c-db2d-4baf-8315-05c651a37dd4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:04.360Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfabf46e-3d69-4394-b93e-32d2b809a063 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:04.360Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfabf46e-3d69-4394-b93e-32d2b809a063 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@61a9b8d 2021-07-13T01:06:04.360Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfabf46e-3d69-4394-b93e-32d2b809a063 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:04.360Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfabf46e-3d69-4394-b93e-32d2b809a063 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:06.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34420d6f-0f42-4f3e-a455-502dcb7d11b1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:06.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34420d6f-0f42-4f3e-a455-502dcb7d11b1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4477f67 2021-07-13T01:06:06.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34420d6f-0f42-4f3e-a455-502dcb7d11b1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:06.060Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34420d6f-0f42-4f3e-a455-502dcb7d11b1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:07.283Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f8fa894-9eb5-4652-9dfc-5071701e8f1a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:07.283Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f8fa894-9eb5-4652-9dfc-5071701e8f1a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2cd43819 2021-07-13T01:06:07.283Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f8fa894-9eb5-4652-9dfc-5071701e8f1a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:07.283Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f8fa894-9eb5-4652-9dfc-5071701e8f1a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:06:07.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:06:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T01:06:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T01:06:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T01:06:09.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92c97176-e378-4475-a5ba-f5c3feefeaf1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:09.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92c97176-e378-4475-a5ba-f5c3feefeaf1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@77f6a423 2021-07-13T01:06:09.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92c97176-e378-4475-a5ba-f5c3feefeaf1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:09.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=92c97176-e378-4475-a5ba-f5c3feefeaf1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:11.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d38915b3-38ed-4a59-baac-6bfeb327161d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:11.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d38915b3-38ed-4a59-baac-6bfeb327161d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@25628fa3 2021-07-13T01:06:11.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d38915b3-38ed-4a59-baac-6bfeb327161d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:11.358Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d38915b3-38ed-4a59-baac-6bfeb327161d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:12.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18ebd327-a580-4b5a-8b35-fcf12b759f92 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:12.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18ebd327-a580-4b5a-8b35-fcf12b759f92 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1a61602f 2021-07-13T01:06:12.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18ebd327-a580-4b5a-8b35-fcf12b759f92 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:12.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=18ebd327-a580-4b5a-8b35-fcf12b759f92 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:14.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=991fd3cd-2533-4314-b1e2-3636153667a3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:14.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=991fd3cd-2533-4314-b1e2-3636153667a3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1778878a 2021-07-13T01:06:14.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=991fd3cd-2533-4314-b1e2-3636153667a3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:14.368Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=991fd3cd-2533-4314-b1e2-3636153667a3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:15.556Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f4ea7cb-47a7-4876-a2af-0d16534682e3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:15.556Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f4ea7cb-47a7-4876-a2af-0d16534682e3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@55edeefa 2021-07-13T01:06:15.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f4ea7cb-47a7-4876-a2af-0d16534682e3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:15.557Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f4ea7cb-47a7-4876-a2af-0d16534682e3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:17.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bdbabf47-40b9-41c5-906e-e44f8695545b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:17.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bdbabf47-40b9-41c5-906e-e44f8695545b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@64436c73 2021-07-13T01:06:17.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bdbabf47-40b9-41c5-906e-e44f8695545b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:17.364Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bdbabf47-40b9-41c5-906e-e44f8695545b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:19.071Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f8ae9ed-7520-45d8-87de-74fe7d957d8e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:19.071Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f8ae9ed-7520-45d8-87de-74fe7d957d8e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@753208cc 2021-07-13T01:06:19.071Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f8ae9ed-7520-45d8-87de-74fe7d957d8e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:19.071Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f8ae9ed-7520-45d8-87de-74fe7d957d8e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:20.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e86013cc-dd16-4aac-a8eb-514b7d2ec987 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:20.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e86013cc-dd16-4aac-a8eb-514b7d2ec987 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@26a58537 2021-07-13T01:06:20.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e86013cc-dd16-4aac-a8eb-514b7d2ec987 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:20.968Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e86013cc-dd16-4aac-a8eb-514b7d2ec987 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:06:22.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:06:24.267Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5e00008-14ed-42be-bd40-dda2b54bdc00 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:24.267Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5e00008-14ed-42be-bd40-dda2b54bdc00 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c052527 2021-07-13T01:06:24.267Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5e00008-14ed-42be-bd40-dda2b54bdc00 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:24.267Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f5e00008-14ed-42be-bd40-dda2b54bdc00 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:24.474Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:06:26.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20f0c2a2-1370-4ab8-9a30-11c8b2437b94 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:26.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20f0c2a2-1370-4ab8-9a30-11c8b2437b94 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63daf220 2021-07-13T01:06:26.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20f0c2a2-1370-4ab8-9a30-11c8b2437b94 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:26.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20f0c2a2-1370-4ab8-9a30-11c8b2437b94 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:29.922Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db7f9345-b0fe-482e-8584-f76c97110354 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:29.923Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db7f9345-b0fe-482e-8584-f76c97110354 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6cbf2ff 2021-07-13T01:06:29.923Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db7f9345-b0fe-482e-8584-f76c97110354 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:29.923Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db7f9345-b0fe-482e-8584-f76c97110354 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:32.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f1b3ccd-56fd-4b61-b51b-4e990ed5629b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:32.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f1b3ccd-56fd-4b61-b51b-4e990ed5629b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@c4d532c 2021-07-13T01:06:32.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f1b3ccd-56fd-4b61-b51b-4e990ed5629b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:32.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8f1b3ccd-56fd-4b61-b51b-4e990ed5629b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:35.571Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36189b96-41b3-46ca-933b-c7615d5818f4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:35.572Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36189b96-41b3-46ca-933b-c7615d5818f4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3954739f 2021-07-13T01:06:35.572Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36189b96-41b3-46ca-933b-c7615d5818f4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:35.572Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=36189b96-41b3-46ca-933b-c7615d5818f4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:06:37.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:06:39.163Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33c3573a-02b9-44be-81e5-672fbdedff38 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:39.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33c3573a-02b9-44be-81e5-672fbdedff38 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@d2bdd67 2021-07-13T01:06:39.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33c3573a-02b9-44be-81e5-672fbdedff38 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:39.164Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=33c3573a-02b9-44be-81e5-672fbdedff38 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:41.767Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02639276-ded5-404a-b28c-98c7c5cdbd2f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:41.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02639276-ded5-404a-b28c-98c7c5cdbd2f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@54ae9a37 2021-07-13T01:06:41.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02639276-ded5-404a-b28c-98c7c5cdbd2f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:41.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=02639276-ded5-404a-b28c-98c7c5cdbd2f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:45.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86b0046e-10d9-4949-8118-8e69f5b4d0b0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:45.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86b0046e-10d9-4949-8118-8e69f5b4d0b0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@329c77bd 2021-07-13T01:06:45.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86b0046e-10d9-4949-8118-8e69f5b4d0b0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:45.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=86b0046e-10d9-4949-8118-8e69f5b4d0b0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:47.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10e38b06-3703-4d44-8a27-ad316e597b5c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:47.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10e38b06-3703-4d44-8a27-ad316e597b5c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@284e22fa 2021-07-13T01:06:47.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10e38b06-3703-4d44-8a27-ad316e597b5c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:47.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=10e38b06-3703-4d44-8a27-ad316e597b5c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:50.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40ae3acc-f7cc-43ac-acf1-e02e56bff330 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:50.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40ae3acc-f7cc-43ac-acf1-e02e56bff330 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7b5d1271 2021-07-13T01:06:50.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40ae3acc-f7cc-43ac-acf1-e02e56bff330 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:50.965Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=40ae3acc-f7cc-43ac-acf1-e02e56bff330 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:06:52.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:06:54.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da3b0c81-e6fe-4313-bff2-771bdcc37ef9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:54.061Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da3b0c81-e6fe-4313-bff2-771bdcc37ef9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5b02400f 2021-07-13T01:06:54.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da3b0c81-e6fe-4313-bff2-771bdcc37ef9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:54.062Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=da3b0c81-e6fe-4313-bff2-771bdcc37ef9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:06:56.818Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a9e7cfc-db77-416e-aaa2-b42860e8c67f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:06:56.818Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a9e7cfc-db77-416e-aaa2-b42860e8c67f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@a72f54d 2021-07-13T01:06:56.819Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a9e7cfc-db77-416e-aaa2-b42860e8c67f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:06:56.819Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9a9e7cfc-db77-416e-aaa2-b42860e8c67f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:07:00.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61aeb086-9d40-4e92-9263-de2fcbc55354 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:07:00.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61aeb086-9d40-4e92-9263-de2fcbc55354 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@53f25f0b 2021-07-13T01:07:00.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61aeb086-9d40-4e92-9263-de2fcbc55354 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:07:00.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=61aeb086-9d40-4e92-9263-de2fcbc55354 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:07:02.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fcc6022-7cb7-4756-a20a-adfe614fb7ee PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:07:02.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fcc6022-7cb7-4756-a20a-adfe614fb7ee PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@40af119e 2021-07-13T01:07:02.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fcc6022-7cb7-4756-a20a-adfe614fb7ee PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:07:02.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5fcc6022-7cb7-4756-a20a-adfe614fb7ee PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:07:06.646Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20a30e17-4381-4caa-9336-1ff4bb92ee70 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:07:06.646Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20a30e17-4381-4caa-9336-1ff4bb92ee70 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@9e423bc 2021-07-13T01:07:06.646Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20a30e17-4381-4caa-9336-1ff4bb92ee70 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:07:06.646Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=20a30e17-4381-4caa-9336-1ff4bb92ee70 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:07:07.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:07:07.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:07:08.678Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13b308a7-c3c3-484f-8898-3f7c10b4ba68 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:07:08.678Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13b308a7-c3c3-484f-8898-3f7c10b4ba68 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@14a16750 2021-07-13T01:07:08.678Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13b308a7-c3c3-484f-8898-3f7c10b4ba68 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:07:08.678Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=13b308a7-c3c3-484f-8898-3f7c10b4ba68 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:07:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T01:07:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T01:07:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T01:07:11.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=30ccb03e-7316-4920-8309-78ebb69c6f6a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:07:11.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=30ccb03e-7316-4920-8309-78ebb69c6f6a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@55ea78c8 2021-07-13T01:07:11.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=30ccb03e-7316-4920-8309-78ebb69c6f6a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:07:11.956Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=30ccb03e-7316-4920-8309-78ebb69c6f6a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:07:14.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a48efba3-4e4c-44d1-8296-de78a4ef02c2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:07:14.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a48efba3-4e4c-44d1-8296-de78a4ef02c2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6df1a00a 2021-07-13T01:07:14.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a48efba3-4e4c-44d1-8296-de78a4ef02c2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:07:14.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a48efba3-4e4c-44d1-8296-de78a4ef02c2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:07:18.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=134bcc8e-937d-4664-bc6e-b74c722906a3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:07:18.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=134bcc8e-937d-4664-bc6e-b74c722906a3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@452087cb 2021-07-13T01:07:18.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=134bcc8e-937d-4664-bc6e-b74c722906a3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:07:18.070Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=134bcc8e-937d-4664-bc6e-b74c722906a3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:07:20.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d88b3d17-1d99-4719-8b20-a66ae6824669 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:07:20.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d88b3d17-1d99-4719-8b20-a66ae6824669 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5a24e3d8 2021-07-13T01:07:20.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d88b3d17-1d99-4719-8b20-a66ae6824669 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:07:20.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d88b3d17-1d99-4719-8b20-a66ae6824669 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:07:22.917Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:07:22.926Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:07:23.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=507441cb-04ac-4ba3-9762-de4221d20d6e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:07:23.769Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=507441cb-04ac-4ba3-9762-de4221d20d6e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5c8efc6c 2021-07-13T01:07:23.769Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=507441cb-04ac-4ba3-9762-de4221d20d6e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:07:23.769Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=507441cb-04ac-4ba3-9762-de4221d20d6e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:07:24.710Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:07:27.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a84064a6-e26a-4ab7-9141-f4b377a99f99 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:07:27.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a84064a6-e26a-4ab7-9141-f4b377a99f99 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3c1fe607 2021-07-13T01:07:27.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a84064a6-e26a-4ab7-9141-f4b377a99f99 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:07:27.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a84064a6-e26a-4ab7-9141-f4b377a99f99 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:07:30.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d25e906b-647c-41c9-ac48-3176b63ba4cd PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:07:30.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d25e906b-647c-41c9-ac48-3176b63ba4cd PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7dc03090 2021-07-13T01:07:30.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d25e906b-647c-41c9-ac48-3176b63ba4cd PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:07:30.156Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d25e906b-647c-41c9-ac48-3176b63ba4cd PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:07:33.164Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=d25e906b-647c-41c9-ac48-3176b63ba4cd ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T01:07:33.165Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=d25e906b-647c-41c9-ac48-3176b63ba4cd ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T01:07:33.165Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=d25e906b-647c-41c9-ac48-3176b63ba4cd ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:07:33.165Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=d25e906b-647c-41c9-ac48-3176b63ba4cd ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:07:35.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5edfa87-6224-41ff-8807-b4d99e433b95 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:07:35.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5edfa87-6224-41ff-8807-b4d99e433b95 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@8726d3 2021-07-13T01:07:35.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5edfa87-6224-41ff-8807-b4d99e433b95 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:07:35.460Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5edfa87-6224-41ff-8807-b4d99e433b95 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:07:36.067Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=e5edfa87-6224-41ff-8807-b4d99e433b95 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:07:36.067Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=e5edfa87-6224-41ff-8807-b4d99e433b95 ErrorCategory=INFO ServiceName=BE Component ON_BOARDING state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:07:36.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5b1e458-6bb1-4225-b4cd-578698b7192f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:07:36.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5b1e458-6bb1-4225-b4cd-578698b7192f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@682d3382 2021-07-13T01:07:36.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5b1e458-6bb1-4225-b4cd-578698b7192f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:07:36.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5b1e458-6bb1-4225-b4cd-578698b7192f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:07:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:07:37.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:07:38.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b398b17-de55-46a4-a0f5-3ec067dd4a92 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:07:38.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b398b17-de55-46a4-a0f5-3ec067dd4a92 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1415daa7 2021-07-13T01:07:38.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b398b17-de55-46a4-a0f5-3ec067dd4a92 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:07:38.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b398b17-de55-46a4-a0f5-3ec067dd4a92 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:07:41.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6bad4ed-8bec-47a0-a345-2cf4ee48e26c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:07:41.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6bad4ed-8bec-47a0-a345-2cf4ee48e26c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d9dc748 2021-07-13T01:07:41.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6bad4ed-8bec-47a0-a345-2cf4ee48e26c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:07:41.772Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d6bad4ed-8bec-47a0-a345-2cf4ee48e26c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:07:45.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=716598fe-3a2d-4b2f-9f33-2847da4c84b9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:07:45.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=716598fe-3a2d-4b2f-9f33-2847da4c84b9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@8c06587 2021-07-13T01:07:45.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=716598fe-3a2d-4b2f-9f33-2847da4c84b9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:07:45.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=716598fe-3a2d-4b2f-9f33-2847da4c84b9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:07:47.961Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8271d828-0a4d-460a-a73d-d209dd3f86b9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:07:47.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8271d828-0a4d-460a-a73d-d209dd3f86b9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@9f0f63e 2021-07-13T01:07:47.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8271d828-0a4d-460a-a73d-d209dd3f86b9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:07:47.962Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8271d828-0a4d-460a-a73d-d209dd3f86b9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:07:50.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0538176d-ed10-4bbf-b887-122cc4a6180a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:07:50.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0538176d-ed10-4bbf-b887-122cc4a6180a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@330ac545 2021-07-13T01:07:50.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0538176d-ed10-4bbf-b887-122cc4a6180a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:07:50.967Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0538176d-ed10-4bbf-b887-122cc4a6180a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:07:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:07:52.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:07:53.568Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e884fe2-1143-48b8-ba26-9270e51bef86 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:07:53.568Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e884fe2-1143-48b8-ba26-9270e51bef86 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@70eef3eb 2021-07-13T01:07:53.568Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e884fe2-1143-48b8-ba26-9270e51bef86 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:07:53.568Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0e884fe2-1143-48b8-ba26-9270e51bef86 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:07:56.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5040aae-366f-4aea-afb8-20eaf7b68fe6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:07:56.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5040aae-366f-4aea-afb8-20eaf7b68fe6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7b095c2c 2021-07-13T01:07:56.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5040aae-366f-4aea-afb8-20eaf7b68fe6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:07:56.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c5040aae-366f-4aea-afb8-20eaf7b68fe6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:07:59.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b4df569-7995-40f9-9534-919b29d103b6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:07:59.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b4df569-7995-40f9-9534-919b29d103b6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@75460475 2021-07-13T01:07:59.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b4df569-7995-40f9-9534-919b29d103b6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:07:59.761Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0b4df569-7995-40f9-9534-919b29d103b6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:08:02.672Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e29d8a04-1233-4198-a063-196e470386df PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:08:02.672Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e29d8a04-1233-4198-a063-196e470386df PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1083d775 2021-07-13T01:08:02.672Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e29d8a04-1233-4198-a063-196e470386df PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:08:02.672Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e29d8a04-1233-4198-a063-196e470386df PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:08:05.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81d54a96-f7ef-4297-9328-7a0bb17ab909 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:08:05.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81d54a96-f7ef-4297-9328-7a0bb17ab909 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@17750f25 2021-07-13T01:08:05.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81d54a96-f7ef-4297-9328-7a0bb17ab909 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:08:05.870Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=81d54a96-f7ef-4297-9328-7a0bb17ab909 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:08:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:08:07.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:08:08.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f42f3313-7da9-4a2c-afe1-9250503101d8 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:08:08.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f42f3313-7da9-4a2c-afe1-9250503101d8 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@262cc9db 2021-07-13T01:08:08.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f42f3313-7da9-4a2c-afe1-9250503101d8 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:08:08.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=f42f3313-7da9-4a2c-afe1-9250503101d8 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:08:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T01:08:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T01:08:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T01:08:11.969Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb2835ab-2980-42c1-8cc1-2cda6749639f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:08:11.969Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb2835ab-2980-42c1-8cc1-2cda6749639f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@638c04c7 2021-07-13T01:08:11.969Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb2835ab-2980-42c1-8cc1-2cda6749639f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:08:11.970Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fb2835ab-2980-42c1-8cc1-2cda6749639f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:08:15.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5d6a37c6-ede7-41f6-8edd-8b2e1fa4be13 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:08:15.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5d6a37c6-ede7-41f6-8edd-8b2e1fa4be13 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6dba5fbc 2021-07-13T01:08:15.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5d6a37c6-ede7-41f6-8edd-8b2e1fa4be13 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:08:15.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5d6a37c6-ede7-41f6-8edd-8b2e1fa4be13 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:08:18.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=058984a4-a33d-4dbc-868e-af1aa4addde4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:08:18.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=058984a4-a33d-4dbc-868e-af1aa4addde4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@52896909 2021-07-13T01:08:18.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=058984a4-a33d-4dbc-868e-af1aa4addde4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:08:18.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=058984a4-a33d-4dbc-868e-af1aa4addde4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:08:20.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=43389301-edf3-4200-8f50-55ca8605c081 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:08:20.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=43389301-edf3-4200-8f50-55ca8605c081 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@12ab0092 2021-07-13T01:08:20.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=43389301-edf3-4200-8f50-55ca8605c081 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:08:20.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=43389301-edf3-4200-8f50-55ca8605c081 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:08:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:08:22.919Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:08:24.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62d5f010-488b-4269-93d2-4d868ec1350c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:08:24.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62d5f010-488b-4269-93d2-4d868ec1350c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1182843c 2021-07-13T01:08:24.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62d5f010-488b-4269-93d2-4d868ec1350c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:08:24.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62d5f010-488b-4269-93d2-4d868ec1350c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:08:25.176Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:08:27.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b868e85a-e2be-4a69-8931-70e757e50bdb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:08:27.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b868e85a-e2be-4a69-8931-70e757e50bdb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2efa7e0f 2021-07-13T01:08:27.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b868e85a-e2be-4a69-8931-70e757e50bdb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:08:27.158Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b868e85a-e2be-4a69-8931-70e757e50bdb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:08:33.234Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6e4f908-5c58-496a-a7cb-98cedf5930dc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:08:33.234Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6e4f908-5c58-496a-a7cb-98cedf5930dc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@37a29bf8 2021-07-13T01:08:33.234Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6e4f908-5c58-496a-a7cb-98cedf5930dc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:08:33.234Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e6e4f908-5c58-496a-a7cb-98cedf5930dc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:08:34.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98c85b38-e72d-4290-8a39-82875f8ec3fc PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:08:34.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98c85b38-e72d-4290-8a39-82875f8ec3fc PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7dc7b20e 2021-07-13T01:08:34.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98c85b38-e72d-4290-8a39-82875f8ec3fc PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:08:34.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=98c85b38-e72d-4290-8a39-82875f8ec3fc PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:08:36.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8de0b6c-da23-413a-b60e-9fc28d60a731 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:08:36.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8de0b6c-da23-413a-b60e-9fc28d60a731 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78617aee 2021-07-13T01:08:36.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8de0b6c-da23-413a-b60e-9fc28d60a731 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:08:36.160Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d8de0b6c-da23-413a-b60e-9fc28d60a731 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:08:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:08:37.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:08:39.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2b4fb28-0d3e-459a-ae0f-ad7d3741b516 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:08:39.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2b4fb28-0d3e-459a-ae0f-ad7d3741b516 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@59679eb 2021-07-13T01:08:39.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2b4fb28-0d3e-459a-ae0f-ad7d3741b516 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:08:39.257Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d2b4fb28-0d3e-459a-ae0f-ad7d3741b516 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:08:41.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a2f2fec-0977-419f-91cb-9cb11d0c90d3 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:08:41.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a2f2fec-0977-419f-91cb-9cb11d0c90d3 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@26484ffa 2021-07-13T01:08:41.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a2f2fec-0977-419f-91cb-9cb11d0c90d3 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:08:41.869Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7a2f2fec-0977-419f-91cb-9cb11d0c90d3 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:08:44.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c759475-602f-40d0-bed5-2448f68ae316 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:08:44.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c759475-602f-40d0-bed5-2448f68ae316 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@665fdeff 2021-07-13T01:08:44.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c759475-602f-40d0-bed5-2448f68ae316 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:08:44.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5c759475-602f-40d0-bed5-2448f68ae316 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:08:47.666Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4fb4aa95-e8e1-4a32-9ab0-a9cac9f4712f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:08:47.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4fb4aa95-e8e1-4a32-9ab0-a9cac9f4712f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@159a6ed0 2021-07-13T01:08:47.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4fb4aa95-e8e1-4a32-9ab0-a9cac9f4712f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:08:47.667Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4fb4aa95-e8e1-4a32-9ab0-a9cac9f4712f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:08:49.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:08:49.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T01:08:50.673Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=4fb4aa95-e8e1-4a32-9ab0-a9cac9f4712f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T01:08:50.674Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=4fb4aa95-e8e1-4a32-9ab0-a9cac9f4712f ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T01:08:50.674Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=4fb4aa95-e8e1-4a32-9ab0-a9cac9f4712f ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:08:50.674Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=4fb4aa95-e8e1-4a32-9ab0-a9cac9f4712f ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:08:52.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:08:52.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:08:52.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:08:54.514Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3ca4768-18b9-4085-a771-0c9e613ef449 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:08:54.514Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3ca4768-18b9-4085-a771-0c9e613ef449 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@23f514a9 2021-07-13T01:08:54.514Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3ca4768-18b9-4085-a771-0c9e613ef449 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:08:54.514Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b3ca4768-18b9-4085-a771-0c9e613ef449 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:08:54.671Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T01:08:54.775Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=b3ca4768-18b9-4085-a771-0c9e613ef449 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:08:54.775Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=b3ca4768-18b9-4085-a771-0c9e613ef449 ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T01:08:55.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34623610-c515-4754-baeb-41dd84edd25f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:08:55.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34623610-c515-4754-baeb-41dd84edd25f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5f963e9f 2021-07-13T01:08:55.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34623610-c515-4754-baeb-41dd84edd25f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:08:55.566Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=34623610-c515-4754-baeb-41dd84edd25f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:08:55.964Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=34623610-c515-4754-baeb-41dd84edd25f ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:08:55.964Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=34623610-c515-4754-baeb-41dd84edd25f ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:08:56.916Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8f05c83-0718-4394-bc04-924a0342331b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:08:56.916Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8f05c83-0718-4394-bc04-924a0342331b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@19bc2753 2021-07-13T01:08:56.916Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8f05c83-0718-4394-bc04-924a0342331b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:08:56.916Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c8f05c83-0718-4394-bc04-924a0342331b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:09:00.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0b1ab86-fe10-4e7f-9902-5493457e20e0 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:09:00.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0b1ab86-fe10-4e7f-9902-5493457e20e0 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@12672891 2021-07-13T01:09:00.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0b1ab86-fe10-4e7f-9902-5493457e20e0 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:09:00.418Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e0b1ab86-fe10-4e7f-9902-5493457e20e0 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:09:03.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3eab3fea-9b1b-4224-b155-9fe534488fca PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:09:03.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3eab3fea-9b1b-4224-b155-9fe534488fca PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@60767147 2021-07-13T01:09:03.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3eab3fea-9b1b-4224-b155-9fe534488fca PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:09:03.058Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3eab3fea-9b1b-4224-b155-9fe534488fca PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:09:05.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01296909-9cf4-414d-9fea-9e118df98e42 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:09:05.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01296909-9cf4-414d-9fea-9e118df98e42 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e7ac910 2021-07-13T01:09:05.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01296909-9cf4-414d-9fea-9e118df98e42 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:09:05.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=01296909-9cf4-414d-9fea-9e118df98e42 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:09:07.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:09:07.459Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T01:09:07.918Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:09:07.925Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:09:08.869Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=01296909-9cf4-414d-9fea-9e118df98e42 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T01:09:08.870Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=01296909-9cf4-414d-9fea-9e118df98e42 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T01:09:08.870Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=01296909-9cf4-414d-9fea-9e118df98e42 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:09:08.870Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=01296909-9cf4-414d-9fea-9e118df98e42 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:09:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T01:09:09.369Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T01:09:09.369Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T01:09:09.663Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T01:09:10.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf00cc78-f106-40a5-9ddf-0e7884afbb64 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:09:10.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf00cc78-f106-40a5-9ddf-0e7884afbb64 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7604e133 2021-07-13T01:09:10.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf00cc78-f106-40a5-9ddf-0e7884afbb64 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:09:10.159Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=bf00cc78-f106-40a5-9ddf-0e7884afbb64 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:09:13.211Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=bf00cc78-f106-40a5-9ddf-0e7884afbb64 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T01:09:13.211Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=bf00cc78-f106-40a5-9ddf-0e7884afbb64 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T01:09:13.211Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=bf00cc78-f106-40a5-9ddf-0e7884afbb64 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:09:13.211Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=bf00cc78-f106-40a5-9ddf-0e7884afbb64 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T01:09:16.721Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8bfcac12-5588-4547-b90b-aeed0371c357 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:09:16.721Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8bfcac12-5588-4547-b90b-aeed0371c357 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@e23c27d 2021-07-13T01:09:16.721Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8bfcac12-5588-4547-b90b-aeed0371c357 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:09:16.721Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8bfcac12-5588-4547-b90b-aeed0371c357 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:09:17.065Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=8bfcac12-5588-4547-b90b-aeed0371c357 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:09:17.065Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=8bfcac12-5588-4547-b90b-aeed0371c357 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:09:18.456Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff0800e9-5238-4367-a45d-01c7d9025c37 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:09:18.456Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff0800e9-5238-4367-a45d-01c7d9025c37 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@715c663 2021-07-13T01:09:18.456Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff0800e9-5238-4367-a45d-01c7d9025c37 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:09:18.456Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ff0800e9-5238-4367-a45d-01c7d9025c37 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:09:20.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4006d1e3-0755-4df4-8d5a-69b6d8012db1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:09:20.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4006d1e3-0755-4df4-8d5a-69b6d8012db1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3941403f 2021-07-13T01:09:20.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4006d1e3-0755-4df4-8d5a-69b6d8012db1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:09:20.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4006d1e3-0755-4df4-8d5a-69b6d8012db1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:09:22.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dba520ed-f4c3-46a8-80fd-5c69254535ec PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:09:22.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dba520ed-f4c3-46a8-80fd-5c69254535ec PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@29e15d40 2021-07-13T01:09:22.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dba520ed-f4c3-46a8-80fd-5c69254535ec PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:09:22.771Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dba520ed-f4c3-46a8-80fd-5c69254535ec PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:09:22.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:09:22.928Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:09:23.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74a02e30-c31e-4dd1-8590-373067deda39 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:09:23.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74a02e30-c31e-4dd1-8590-373067deda39 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3274ee34 2021-07-13T01:09:23.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74a02e30-c31e-4dd1-8590-373067deda39 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:09:23.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=74a02e30-c31e-4dd1-8590-373067deda39 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:09:26.374Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:09:26.874Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=387a3bd4-4074-4d84-a8b0-bb891d9bdf89 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:09:26.874Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=387a3bd4-4074-4d84-a8b0-bb891d9bdf89 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1b7b10c7 2021-07-13T01:09:26.874Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=387a3bd4-4074-4d84-a8b0-bb891d9bdf89 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:09:26.874Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=387a3bd4-4074-4d84-a8b0-bb891d9bdf89 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:09:30.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51b7eaf0-79e1-43f4-9980-320f0d2108a1 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:09:30.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51b7eaf0-79e1-43f4-9980-320f0d2108a1 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2808eb2f 2021-07-13T01:09:30.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51b7eaf0-79e1-43f4-9980-320f0d2108a1 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:09:30.563Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=51b7eaf0-79e1-43f4-9980-320f0d2108a1 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:09:32.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c8cba95-6f52-40f9-8906-6d00c0c98384 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:09:32.974Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c8cba95-6f52-40f9-8906-6d00c0c98384 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@ba0b833 2021-07-13T01:09:32.974Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c8cba95-6f52-40f9-8906-6d00c0c98384 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:09:32.974Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=2c8cba95-6f52-40f9-8906-6d00c0c98384 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:09:35.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5779b770-e86c-4663-a0a6-89347ff7e501 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:09:35.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5779b770-e86c-4663-a0a6-89347ff7e501 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1e30e23d 2021-07-13T01:09:35.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5779b770-e86c-4663-a0a6-89347ff7e501 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:09:35.763Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5779b770-e86c-4663-a0a6-89347ff7e501 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:09:37.913Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:09:37.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:09:38.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=876bf164-e4d9-409d-b3ed-2a189e0f5b8b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:09:38.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=876bf164-e4d9-409d-b3ed-2a189e0f5b8b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5d0e3414 2021-07-13T01:09:38.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=876bf164-e4d9-409d-b3ed-2a189e0f5b8b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:09:38.858Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=876bf164-e4d9-409d-b3ed-2a189e0f5b8b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:09:45.317Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a7b4261-eb48-4e84-b4e1-165c0c0ce7a6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:09:45.317Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a7b4261-eb48-4e84-b4e1-165c0c0ce7a6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3230ed09 2021-07-13T01:09:45.317Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a7b4261-eb48-4e84-b4e1-165c0c0ce7a6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:09:45.317Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0a7b4261-eb48-4e84-b4e1-165c0c0ce7a6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:09:46.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:09:46.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend probably lost connectivity to JanusGraph DB. Please check the logs for more information. 2021-07-13T01:09:48.324Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR RequestID=0a7b4261-eb48-4e84-b4e1-165c0c0ce7a6 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=ExecuteRestRequest Internal flow error. Operation: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out 2021-07-13T01:09:48.324Z [BE-Health-Check-Task] ERROR o.o.s.b.c.h.HealthCheckBusinessLogic RequestID=0a7b4261-eb48-4e84-b4e1-165c0c0ce7a6 ErrorCategory=ERROR ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=500 PartnerName=SDC.BE TargetEntity=null ON_BOARDING unexpected response: org.openecomp.sdc.common.http.client.api.HttpExecuteException: Execute request GET https://sdc-onboarding-be.onap:8445/onboarding-api/v1.0/healthcheck HTTP/1.1 failed with exception: Read timed out at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:124) at org.openecomp.sdc.common.http.client.api.HttpClient.get(HttpClient.java:66) at org.openecomp.sdc.common.http.client.api.HttpRequestHandler.get(HttpRequestHandler.java:78) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:51) at org.openecomp.sdc.common.http.client.api.HttpRequest.get(HttpRequest.java:47) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getHostedComponentsBeHealthCheck(HealthCheckBusinessLogic.java:232) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic.getBeHealthCheckInfos(HealthCheckBusinessLogic.java:145) at org.openecomp.sdc.be.components.health.HealthCheckBusinessLogic$HealthCheckScheduledTask.run(HealthCheckBusinessLogic.java:405) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketTimeoutException: Read timed out at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/java.net.SocketInputStream.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.openecomp.sdc.common.http.client.api.HttpClient.execute(HttpClient.java:118) ... 13 common frames omitted 2021-07-13T01:09:48.324Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=0a7b4261-eb48-4e84-b4e1-165c0c0ce7a6 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component ON_BOARDING state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:09:48.324Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=0a7b4261-eb48-4e84-b4e1-165c0c0ce7a6 ErrorCategory=INFO ServiceName=ExecuteRestRequest TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:09:49.458Z [JanusGraph-Health-Check-Task] ERROR o.o.s.b.d.j.JanusGraphClient RequestID=null ErrorCategory=ERROR ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck JanusGraph Health Check Failed. java.util.concurrent.TimeoutException 2021-07-13T01:09:50.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f106a5a-c28c-4d9e-99f3-86f398c9654b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:09:50.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f106a5a-c28c-4d9e-99f3-86f398c9654b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3cdfba5c 2021-07-13T01:09:50.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f106a5a-c28c-4d9e-99f3-86f398c9654b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:09:50.515Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1f106a5a-c28c-4d9e-99f3-86f398c9654b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:09:50.921Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=1f106a5a-c28c-4d9e-99f3-86f398c9654b ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from UP to DOWN ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:09:50.921Z [BE-Health-Check-Task] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=CRITICAL RequestID=1f106a5a-c28c-4d9e-99f3-86f398c9654b ErrorCategory=ERROR ServiceName=BE Component JANUSGRAPH state changed from UP to DOWN TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend probably lost connectivity to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. Please check the logs for more information. 2021-07-13T01:09:51.859Z [JanusGraph-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=null ErrorCategory=INFO ServerIPAddress=10.242.98.212 ServiceName=janusgraphHealthCheck TargetServiceName=null ErrorCode=200 ServerFQDN=dev-sdc-be-7dd68c74d5-m8mrp TargetEntity=janusgraphHealthCheck ASDC Backend connection recovery to JanusGraph DB. 2021-07-13T01:09:51.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea0c062f-d8b0-4a85-a574-efc247f72488 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:09:51.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea0c062f-d8b0-4a85-a574-efc247f72488 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@965b8dc 2021-07-13T01:09:51.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea0c062f-d8b0-4a85-a574-efc247f72488 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:09:51.964Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ea0c062f-d8b0-4a85-a574-efc247f72488 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:09:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:09:52.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:09:53.471Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=546fc0d2-2812-40d3-9676-4a1afb3fa586 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:09:53.471Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=546fc0d2-2812-40d3-9676-4a1afb3fa586 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3c08445c 2021-07-13T01:09:53.471Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=546fc0d2-2812-40d3-9676-4a1afb3fa586 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:09:53.471Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=546fc0d2-2812-40d3-9676-4a1afb3fa586 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:09:54.262Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=546fc0d2-2812-40d3-9676-4a1afb3fa586 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=BE Component JANUSGRAPH state changed from DOWN to UP ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:09:54.262Z [BE-Health-Check-Task] INFO o.o.s.c.config.EcompErrorLogUtil alarmSeverity=INFORMATIONAL RequestID=546fc0d2-2812-40d3-9676-4a1afb3fa586 ErrorCategory=INFO ServiceName=BE Component JANUSGRAPH state changed from DOWN to UP TargetServiceName=null ErrorCode=200 PartnerName=SDC.BE TargetEntity=beHealthCheck ASDC Backend Recovery to either one of the following components: JanusGraph DB, Cassandra, Onboarding, UEB Cluster. 2021-07-13T01:09:55.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db410eff-65d9-49b6-9386-db69d611fa1c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:09:55.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db410eff-65d9-49b6-9386-db69d611fa1c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7dc414db 2021-07-13T01:09:55.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db410eff-65d9-49b6-9386-db69d611fa1c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:09:55.161Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=db410eff-65d9-49b6-9386-db69d611fa1c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:09:56.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7329d1e-e97f-4aa6-a727-4c86f2009965 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:09:56.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7329d1e-e97f-4aa6-a727-4c86f2009965 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@648dcd85 2021-07-13T01:09:56.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7329d1e-e97f-4aa6-a727-4c86f2009965 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:09:56.665Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e7329d1e-e97f-4aa6-a727-4c86f2009965 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:09:59.769Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b68ddd9-6a71-4bac-b5ea-2103724eea00 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:09:59.769Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b68ddd9-6a71-4bac-b5ea-2103724eea00 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@400a3a90 2021-07-13T01:09:59.769Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b68ddd9-6a71-4bac-b5ea-2103724eea00 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:09:59.769Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7b68ddd9-6a71-4bac-b5ea-2103724eea00 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:10:02.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26faa722-2c86-49d4-bf84-157af313e46c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:10:02.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26faa722-2c86-49d4-bf84-157af313e46c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3e5ebc69 2021-07-13T01:10:02.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26faa722-2c86-49d4-bf84-157af313e46c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:10:02.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=26faa722-2c86-49d4-bf84-157af313e46c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:10:05.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a962ec6c-13c2-44cb-9a90-977864885260 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:10:05.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a962ec6c-13c2-44cb-9a90-977864885260 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@63e115b5 2021-07-13T01:10:05.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a962ec6c-13c2-44cb-9a90-977864885260 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:10:05.757Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a962ec6c-13c2-44cb-9a90-977864885260 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:10:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:10:07.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:10:08.663Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7efed8d5-3cac-42ad-8db0-78591d05bc79 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:10:08.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7efed8d5-3cac-42ad-8db0-78591d05bc79 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@690a0da9 2021-07-13T01:10:08.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7efed8d5-3cac-42ad-8db0-78591d05bc79 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:10:08.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=7efed8d5-3cac-42ad-8db0-78591d05bc79 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:10:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T01:10:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T01:10:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T01:10:11.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f2f814c-468a-413c-b5f4-0d0c25e7f98b PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:10:11.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f2f814c-468a-413c-b5f4-0d0c25e7f98b PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@78c574a2 2021-07-13T01:10:11.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f2f814c-468a-413c-b5f4-0d0c25e7f98b PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:10:11.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f2f814c-468a-413c-b5f4-0d0c25e7f98b PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:10:14.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ecd01fe2-0ff2-4007-a41d-076314fe34b5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:10:14.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ecd01fe2-0ff2-4007-a41d-076314fe34b5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@33b32227 2021-07-13T01:10:14.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ecd01fe2-0ff2-4007-a41d-076314fe34b5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:10:14.959Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ecd01fe2-0ff2-4007-a41d-076314fe34b5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:10:17.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abbdd91d-c9d3-4eba-a2e9-26f8aa5a7e98 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:10:17.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abbdd91d-c9d3-4eba-a2e9-26f8aa5a7e98 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@5814d559 2021-07-13T01:10:17.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abbdd91d-c9d3-4eba-a2e9-26f8aa5a7e98 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:10:17.856Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=abbdd91d-c9d3-4eba-a2e9-26f8aa5a7e98 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:10:20.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3dbbcd64-0bd4-447c-a1d8-3352d366e987 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:10:20.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3dbbcd64-0bd4-447c-a1d8-3352d366e987 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@61caecb7 2021-07-13T01:10:20.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3dbbcd64-0bd4-447c-a1d8-3352d366e987 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:10:20.860Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3dbbcd64-0bd4-447c-a1d8-3352d366e987 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:10:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:10:22.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:10:23.730Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa059557-31d5-46a8-87c1-4b7240a5650d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:10:23.730Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa059557-31d5-46a8-87c1-4b7240a5650d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3279e448 2021-07-13T01:10:23.730Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa059557-31d5-46a8-87c1-4b7240a5650d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:10:23.730Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=fa059557-31d5-46a8-87c1-4b7240a5650d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:10:25.393Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:10:26.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c63e130f-3c27-411a-9032-be3634825360 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:10:26.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c63e130f-3c27-411a-9032-be3634825360 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@18714a3a 2021-07-13T01:10:26.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c63e130f-3c27-411a-9032-be3634825360 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:10:26.558Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c63e130f-3c27-411a-9032-be3634825360 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:10:28.318Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0tz8msqbei0cg23k3cfwowhyd344 2021-07-13T01:10:28.318Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node09q36a5ev3lc05eiilbghrpj8345 2021-07-13T01:10:28.318Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01xm6r72uv5be41emb6ve50adop339 2021-07-13T01:10:28.318Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01wzi9kib5bebr1a2qi0mq7tefp338 2021-07-13T01:10:28.318Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node018bv4q7fi0gnemr5fpaeoadr346 2021-07-13T01:10:28.318Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0k57n6zx9gpmtacn3uc2qhef1340 2021-07-13T01:10:28.318Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node08x751blh8ikefi2f6p6azx337 2021-07-13T01:10:28.318Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0frpkd7an6o2w11wejllwvut45343 2021-07-13T01:10:28.318Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node0c0q1dj572cmljjoea0j4dmwh341 2021-07-13T01:10:28.318Z [Scheduler-7967307-1] INFO o.o.p.c.o.l.PortalTimeoutHandler Session getting destroyed - id: node01tionchv2tv72dopo8sxjku0a342 2021-07-13T01:10:30.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f7b5bd5-3eb8-4b3a-a8a5-8eede82e8ceb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:10:30.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f7b5bd5-3eb8-4b3a-a8a5-8eede82e8ceb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@66b1ea70 2021-07-13T01:10:30.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f7b5bd5-3eb8-4b3a-a8a5-8eede82e8ceb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:10:30.063Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0f7b5bd5-3eb8-4b3a-a8a5-8eede82e8ceb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:10:32.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1cf61e80-b529-4994-bab3-16a95f73498d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:10:32.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1cf61e80-b529-4994-bab3-16a95f73498d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@50281d45 2021-07-13T01:10:32.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1cf61e80-b529-4994-bab3-16a95f73498d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:10:32.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=1cf61e80-b529-4994-bab3-16a95f73498d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:10:35.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32a2c5fe-9387-4595-a80b-b138a5828d89 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:10:35.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32a2c5fe-9387-4595-a80b-b138a5828d89 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2d0ee8ac 2021-07-13T01:10:35.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32a2c5fe-9387-4595-a80b-b138a5828d89 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:10:35.859Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=32a2c5fe-9387-4595-a80b-b138a5828d89 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:10:37.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:10:37.922Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:10:38.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f259b97-4353-46e2-a1f1-16a2e0b06c61 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:10:38.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f259b97-4353-46e2-a1f1-16a2e0b06c61 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3fbc57ad 2021-07-13T01:10:38.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f259b97-4353-46e2-a1f1-16a2e0b06c61 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:10:38.760Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f259b97-4353-46e2-a1f1-16a2e0b06c61 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:10:41.872Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ed8b939-6314-4bfa-9e53-622a9c1e33e6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:10:41.872Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ed8b939-6314-4bfa-9e53-622a9c1e33e6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@56b7c840 2021-07-13T01:10:41.872Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ed8b939-6314-4bfa-9e53-622a9c1e33e6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:10:41.872Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=0ed8b939-6314-4bfa-9e53-622a9c1e33e6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:10:44.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4aea6fae-dd85-497e-86ec-103698a70b3e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:10:44.862Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4aea6fae-dd85-497e-86ec-103698a70b3e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@466535e6 2021-07-13T01:10:44.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4aea6fae-dd85-497e-86ec-103698a70b3e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:10:44.863Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4aea6fae-dd85-497e-86ec-103698a70b3e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:10:47.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfe595fd-8339-4cee-9c12-84e4bdb612a9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:10:47.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfe595fd-8339-4cee-9c12-84e4bdb612a9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1c74fe0e 2021-07-13T01:10:47.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfe595fd-8339-4cee-9c12-84e4bdb612a9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:10:47.768Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=dfe595fd-8339-4cee-9c12-84e4bdb612a9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:10:50.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d2aa7db-0c50-42f1-9ff7-abf0254d3e0e PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:10:50.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d2aa7db-0c50-42f1-9ff7-abf0254d3e0e PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4128c312 2021-07-13T01:10:50.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d2aa7db-0c50-42f1-9ff7-abf0254d3e0e PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:10:50.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=3d2aa7db-0c50-42f1-9ff7-abf0254d3e0e PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:10:52.917Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:10:52.926Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:10:53.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e4bc186-9f5f-4c06-a2b5-4f4c34011dfa PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:10:53.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e4bc186-9f5f-4c06-a2b5-4f4c34011dfa PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2935f6bf 2021-07-13T01:10:53.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e4bc186-9f5f-4c06-a2b5-4f4c34011dfa PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:10:53.759Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e4bc186-9f5f-4c06-a2b5-4f4c34011dfa PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:10:56.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cee9a6a2-311f-4777-871d-12c9b17d9319 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:10:56.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cee9a6a2-311f-4777-871d-12c9b17d9319 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@46902c7a 2021-07-13T01:10:56.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cee9a6a2-311f-4777-871d-12c9b17d9319 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:10:56.764Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=cee9a6a2-311f-4777-871d-12c9b17d9319 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:10:59.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5374b8b5-1284-4fcb-a7e4-a78f66055b5c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:10:59.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5374b8b5-1284-4fcb-a7e4-a78f66055b5c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7daf3e28 2021-07-13T01:10:59.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5374b8b5-1284-4fcb-a7e4-a78f66055b5c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:10:59.973Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5374b8b5-1284-4fcb-a7e4-a78f66055b5c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:11:02.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b74fd8dc-5c4c-4700-a254-b8e068f9d20f PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:11:02.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b74fd8dc-5c4c-4700-a254-b8e068f9d20f PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7a9852dc 2021-07-13T01:11:02.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b74fd8dc-5c4c-4700-a254-b8e068f9d20f PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:11:02.864Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=b74fd8dc-5c4c-4700-a254-b8e068f9d20f PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:11:06.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aaa55c42-52ac-43bd-9959-69588ee9b4d9 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:11:06.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aaa55c42-52ac-43bd-9959-69588ee9b4d9 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7c1ed33e 2021-07-13T01:11:06.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aaa55c42-52ac-43bd-9959-69588ee9b4d9 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:11:06.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=aaa55c42-52ac-43bd-9959-69588ee9b4d9 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:11:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:11:07.920Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:11:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T01:11:09.368Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T01:11:09.368Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T01:11:11.021Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=77a74689-373f-4e5a-ba0d-3a388f3236b2 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:11:11.021Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=77a74689-373f-4e5a-ba0d-3a388f3236b2 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3baad33b 2021-07-13T01:11:11.021Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=77a74689-373f-4e5a-ba0d-3a388f3236b2 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:11:11.021Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=77a74689-373f-4e5a-ba0d-3a388f3236b2 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:11:12.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eceb2dc3-414f-49ee-a715-c44586766762 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:11:12.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eceb2dc3-414f-49ee-a715-c44586766762 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3ff0f418 2021-07-13T01:11:12.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eceb2dc3-414f-49ee-a715-c44586766762 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:11:12.668Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eceb2dc3-414f-49ee-a715-c44586766762 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:11:15.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47c97daa-c0e0-4d89-90af-c614a62dd413 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:11:15.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47c97daa-c0e0-4d89-90af-c614a62dd413 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7464f88 2021-07-13T01:11:15.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47c97daa-c0e0-4d89-90af-c614a62dd413 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:11:15.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=47c97daa-c0e0-4d89-90af-c614a62dd413 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:11:17.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f858981-6ced-419c-b945-71bfb3334813 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:11:17.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f858981-6ced-419c-b945-71bfb3334813 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@d4bfb7d 2021-07-13T01:11:17.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f858981-6ced-419c-b945-71bfb3334813 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:11:17.958Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=4f858981-6ced-419c-b945-71bfb3334813 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:11:20.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8384cf25-a495-4a8e-a878-ba57e1e6bf7a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:11:20.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8384cf25-a495-4a8e-a878-ba57e1e6bf7a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6015c2e4 2021-07-13T01:11:20.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8384cf25-a495-4a8e-a878-ba57e1e6bf7a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:11:20.570Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=8384cf25-a495-4a8e-a878-ba57e1e6bf7a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:11:22.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:11:22.923Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:11:23.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=290f4be4-2be5-4da8-b20a-3db04a4cdb24 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:11:23.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=290f4be4-2be5-4da8-b20a-3db04a4cdb24 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@58fdaaca 2021-07-13T01:11:23.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=290f4be4-2be5-4da8-b20a-3db04a4cdb24 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:11:23.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=290f4be4-2be5-4da8-b20a-3db04a4cdb24 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:11:26.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=671bc6cb-4e1f-440b-98d3-067c537a95f5 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:11:26.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=671bc6cb-4e1f-440b-98d3-067c537a95f5 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6d728452 2021-07-13T01:11:26.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=671bc6cb-4e1f-440b-98d3-067c537a95f5 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:11:26.966Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=671bc6cb-4e1f-440b-98d3-067c537a95f5 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:11:30.020Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a82c381e-bc79-43cb-92b8-c53aaaaa313a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:11:30.021Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a82c381e-bc79-43cb-92b8-c53aaaaa313a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3fb00a92 2021-07-13T01:11:30.021Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a82c381e-bc79-43cb-92b8-c53aaaaa313a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:11:30.021Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a82c381e-bc79-43cb-92b8-c53aaaaa313a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:11:30.133Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:11:32.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c384ab15-8779-4f81-8c50-b4e50a6675cf PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:11:32.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c384ab15-8779-4f81-8c50-b4e50a6675cf PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@717ced62 2021-07-13T01:11:32.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c384ab15-8779-4f81-8c50-b4e50a6675cf PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:11:32.765Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=c384ab15-8779-4f81-8c50-b4e50a6675cf PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:11:36.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e155bc1-6168-48fe-9fc3-3f30995f95db PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:11:36.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e155bc1-6168-48fe-9fc3-3f30995f95db PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@44e2a27d 2021-07-13T01:11:36.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e155bc1-6168-48fe-9fc3-3f30995f95db PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:11:36.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5e155bc1-6168-48fe-9fc3-3f30995f95db PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:11:37.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:11:37.924Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:11:39.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d4fe8700-69b3-43d4-9c67-ba22b2db1db6 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:11:39.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d4fe8700-69b3-43d4-9c67-ba22b2db1db6 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3d4bfea7 2021-07-13T01:11:39.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d4fe8700-69b3-43d4-9c67-ba22b2db1db6 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:11:39.067Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d4fe8700-69b3-43d4-9c67-ba22b2db1db6 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:11:41.564Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eb31c352-bfbd-4885-bbb7-c227a75313ea PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:11:41.564Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eb31c352-bfbd-4885-bbb7-c227a75313ea PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@2108706 2021-07-13T01:11:41.564Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eb31c352-bfbd-4885-bbb7-c227a75313ea PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:11:41.564Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=eb31c352-bfbd-4885-bbb7-c227a75313ea PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:11:45.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b7b2859-7723-43d1-aa1a-126fe67ef0eb PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:11:45.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b7b2859-7723-43d1-aa1a-126fe67ef0eb PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@14117ddb 2021-07-13T01:11:45.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b7b2859-7723-43d1-aa1a-126fe67ef0eb PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:11:45.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=5b7b2859-7723-43d1-aa1a-126fe67ef0eb PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:11:48.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df1f7dac-7931-48df-a6e8-27788d6468c4 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:11:48.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df1f7dac-7931-48df-a6e8-27788d6468c4 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4531d46e 2021-07-13T01:11:48.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df1f7dac-7931-48df-a6e8-27788d6468c4 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:11:48.256Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=df1f7dac-7931-48df-a6e8-27788d6468c4 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:11:51.056Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e8ca95af-80e8-4f34-a02b-dded68b7722a PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:11:51.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e8ca95af-80e8-4f34-a02b-dded68b7722a PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@27eda4a9 2021-07-13T01:11:51.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e8ca95af-80e8-4f34-a02b-dded68b7722a PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:11:51.057Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e8ca95af-80e8-4f34-a02b-dded68b7722a PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:11:52.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:11:52.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:11:53.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adff2615-304d-45ff-bbcb-57057badbf62 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:11:53.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adff2615-304d-45ff-bbcb-57057badbf62 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@21c33fe5 2021-07-13T01:11:53.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adff2615-304d-45ff-bbcb-57057badbf62 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:11:53.457Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=adff2615-304d-45ff-bbcb-57057badbf62 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:11:56.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3b028ca-6291-43b3-bdc6-29a0ac51b9ff PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:11:56.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3b028ca-6291-43b3-bdc6-29a0ac51b9ff PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3e2361a3 2021-07-13T01:11:56.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3b028ca-6291-43b3-bdc6-29a0ac51b9ff PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:11:56.762Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=d3b028ca-6291-43b3-bdc6-29a0ac51b9ff PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:12:00.627Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=694b002a-d92b-4bde-9fc3-8792a3e13d9c PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:12:00.627Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=694b002a-d92b-4bde-9fc3-8792a3e13d9c PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@757a50ad 2021-07-13T01:12:00.627Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=694b002a-d92b-4bde-9fc3-8792a3e13d9c PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:12:00.627Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=694b002a-d92b-4bde-9fc3-8792a3e13d9c PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:12:02.567Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e982cf56-af4d-42c4-aa2e-4cf436c01423 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:12:02.567Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e982cf56-af4d-42c4-aa2e-4cf436c01423 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@3ff86a47 2021-07-13T01:12:02.567Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e982cf56-af4d-42c4-aa2e-4cf436c01423 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:12:02.567Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e982cf56-af4d-42c4-aa2e-4cf436c01423 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:12:06.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75d0a137-755f-47be-85a4-51c7caf01fb7 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:12:06.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75d0a137-755f-47be-85a4-51c7caf01fb7 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7fc564a 2021-07-13T01:12:06.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75d0a137-755f-47be-85a4-51c7caf01fb7 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:12:06.059Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=75d0a137-755f-47be-85a4-51c7caf01fb7 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:12:07.914Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:12:07.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:12:09.015Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b5c7814-744b-4b0b-9280-a85c5b54160d PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:12:09.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b5c7814-744b-4b0b-9280-a85c5b54160d PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@22f7116f 2021-07-13T01:12:09.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b5c7814-744b-4b0b-9280-a85c5b54160d PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:12:09.016Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=9b5c7814-744b-4b0b-9280-a85c5b54160d PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:12:09.367Z [TopicPollingThread-1] INFO c.a.n.c.c.impl.CambriaConsumerImpl UEB GET /events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 2021-07-13T01:12:09.369Z [TopicPollingThread-1] WARN c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 will send credentials over a clear channel. 2021-07-13T01:12:09.369Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/events/SDC-DISTR-STATUS-TOPIC-AUTO/sdc-AUTO/sdc-AUTO1?timeout=15000 (as iPIxkpAMI8qTcQj8) ... 2021-07-13T01:12:11.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a95f4459-9b8b-4811-af0e-ee3da9449166 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:12:11.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a95f4459-9b8b-4811-af0e-ee3da9449166 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@52fc2a4 2021-07-13T01:12:11.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a95f4459-9b8b-4811-af0e-ee3da9449166 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:12:11.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a95f4459-9b8b-4811-af0e-ee3da9449166 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:12:14.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62ae9293-d921-4961-ab1c-40465bfe8b80 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:12:14.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62ae9293-d921-4961-ab1c-40465bfe8b80 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@79f92a14 2021-07-13T01:12:14.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62ae9293-d921-4961-ab1c-40465bfe8b80 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:12:14.664Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=62ae9293-d921-4961-ab1c-40465bfe8b80 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:12:17.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5cc03b4-bf29-4010-9c0a-68c8675e70ab PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:12:17.866Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5cc03b4-bf29-4010-9c0a-68c8675e70ab PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@1cd27d67 2021-07-13T01:12:17.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5cc03b4-bf29-4010-9c0a-68c8675e70ab PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:12:17.867Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e5cc03b4-bf29-4010-9c0a-68c8675e70ab PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:12:20.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca589c55-22d4-4c7f-8c21-3a5f24758280 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:12:20.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca589c55-22d4-4c7f-8c21-3a5f24758280 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@6b47bfb 2021-07-13T01:12:20.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca589c55-22d4-4c7f-8c21-3a5f24758280 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:12:20.960Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=ca589c55-22d4-4c7f-8c21-3a5f24758280 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:12:22.915Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient GET http://message-router.onap:3904/apiKeys/iPIxkpAMI8qTcQj8 (anonymous) ... 2021-07-13T01:12:22.921Z [UEB-Health-Check-Thread] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:12:24.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6345fd3-0ab2-48e2-9f01-2894f1ac6b52 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:12:24.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6345fd3-0ab2-48e2-9f01-2894f1ac6b52 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@7d75b36b 2021-07-13T01:12:24.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6345fd3-0ab2-48e2-9f01-2894f1ac6b52 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:12:24.064Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=a6345fd3-0ab2-48e2-9f01-2894f1ac6b52 PartnerName=SDC.BE The cassandra down nodes number is 0 2021-07-13T01:12:25.541Z [TopicPollingThread-1] INFO c.att.nsa.apiClient.http.HttpClient \t--> HTTP/1.1 200 OK 2021-07-13T01:12:26.654Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e81c5ca0-2b1c-4ceb-8163-a4c2a9240737 PartnerName=SDC.BE creating cluster for Cassandra for monitoring. 2021-07-13T01:12:26.654Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e81c5ca0-2b1c-4ceb-8163-a4c2a9240737 PartnerName=SDC.BE The cassandra session is com.datastax.driver.core.SessionManager@4dec9f5b 2021-07-13T01:12:26.654Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e81c5ca0-2b1c-4ceb-8163-a4c2a9240737 PartnerName=SDC.BE The number of cassandra nodes is:3 2021-07-13T01:12:26.654Z [BE-Health-Check-Task] INFO o.o.s.b.c.impl.CassandraHealthCheck RequestID=e81c5ca0-2b1c-4ceb-8163-a4c2a9240737 PartnerName=SDC.BE The cassandra down nodes number is 0